arnaucube.com/public/flock-botnet.html
public/flock-botnet.htmlRaw
1<!DOCTYPE html>
2<html lang="en">
3
4 <head>
5 <meta name="description" content="Webpage description goes here" />
6 <meta charset="utf-8">
7 <title>Auto generated tweets from Markov chains - ArnauCube - Blog</title>
8 <meta name="viewport" content="width=device-width, initial-scale=1">
9 <meta name="author" content="">
10
11 <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous">
12 <link rel="stylesheet" href="css/style.css">
13
14 <!-- highlightjs -->
15 <!-- <link rel="stylesheet" href="js/highlightjs/atom-one-dark.css"> -->
16 <link rel="stylesheet" href="js/highlightjs/gruvbox-dark.css">
17 <script src="js/highlightjs/highlight.pack.js"></script>
18
19 <!-- katex -->
20 <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.13.11/dist/katex.min.css" integrity="sha384-Um5gpz1odJg5Z4HAmzPtgZKdTBHZdw8S29IecapCSB31ligYPhHQZMIlWLYQGVoc" crossorigin="anonymous">
21 </head>
22
23 <body>
24
25 <!-- o_gradient_background" -->
26 <nav id="mainNav" class="navbar navbar-default navbar-fixed-top"
27 style="height:50px;font-size:130%;">
28 <div class="container">
29 <a href="/blog" style="color:#000;">Blog index</a>
30 <a href="/" style="color:#000;float:right;">arnaucube.com</a>
31 </div>
32 </nav>
33 <div class="o_gradient_background" style="height:5px;"></div>
34
35
36
37 <div class="container" style="margin-top:80px;max-width:800px;">
38 <h1>Auto generated tweets from Markov chains</h1>
39
40<p><em>2017-12-29</em></p>
41
42<p>The main goal of this article is to explain how to develop a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains. As this is a project to learn, we have made everything from scratch.</p>
43
44<p>The idea of mixing twitter bots with Markov chains was in a Twitter conversation with <a href="https://twitter.com/x0rz">@x0rz</a></p>
45
46<h2>1. Markov chains</h2>
47
48<p>A Markov chain is a sequence of stochastic events (based on probabilities) where the current state of a variable or system is independent of all past states, except the current state.</p>
49
50<p><a href="https://en.wikipedia.org/wiki/Markov_chain">https://en.wikipedia.org/wiki/Markov_chain</a></p>
51
52<p>In our case, we will use Markov chains to analyze the probabilities that after a word comes another concrete word. So, we will generate some diagram like the following one, but with thousands of words.</p>
53
54<p><img src="img/posts/flock-botnet/markovchain.png" alt="flock-botnet" title="Markov chain" /></p>
55
56<p>In our case, we need as input some text document with thousands of words, to get a better input data. In this example we have made it with the book “The Critique of Pure Reason”, by Immanuel Kant (<a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">http://www.gutenberg.org/cache/epub/4280/pg4280.txt</a>), just because is the first book that we have found in .txt format.</p>
57
58<h3>1.1 Calculating the Markov chains</h3>
59
60<p>First we need to read the text file:</p>
61
62<pre><code class="language-go">func readTxt(path string) (string, error) {
63 data, err := ioutil.ReadFile(path)
64 if err != nil {
65 //Do something
66 }
67 dataClean := strings.Replace(string(data), "\n", " ", -1)
68 content := string(dataClean)
69 return content, err
70}
71</code></pre>
72
73<p>To calculate the probabilities of the Markov states, we have made the following function that analyzes the full input text, and stores the Markov states:</p>
74
75<pre><code class="language-go">func calcMarkovStates(words []string) []State {
76 var states []State
77 //count words
78 for i := 0; i < len(words)-1; i++ {
79 var iState int
80 states, iState = addWordToStates(states, words[i])
81 if iState < len(words) {
82 states[iState].NextStates, _ = addWordToStates(states[iState].NextStates, words[i+1])
83 }
84
85 printLoading(i, len(words))
86 }
87
88 //count prob
89 for i := 0; i < len(states); i++ {
90 states[i].Prob = (float64(states[i].Count) / float64(len(words)) * 100)
91 for j := 0; j < len(states[i].NextStates); j++ {
92 states[i].NextStates[j].Prob = (float64(states[i].NextStates[j].Count) / float64(len(words)) * 100)
93 }
94 }
95 fmt.Println("\ntotal words computed: " + strconv.Itoa(len(words)))
96 return states
97}
98</code></pre>
99
100<p>The <em>printLoading</em> function, is just a simple function to print in the terminal the % of the process done:</p>
101
102<pre><code class="language-go">func printLoading(n int, total int) {
103 var bar []string
104 tantPerFourty := int((float64(n) / float64(total)) * 40)
105 tantPerCent := int((float64(n) / float64(total)) * 100)
106 for i := 0; i < tantPerFourty; i++ {
107 bar = append(bar, "█")
108 }
109 progressBar := strings.Join(bar, "")
110 fmt.Printf("\r " + progressBar + " - " + strconv.Itoa(tantPerCent) + "")
111}
112</code></pre>
113
114<p><img src="img/posts/flock-botnet/progressbarMarkov.gif" alt="flock-botnet" title="Markov chain" /></p>
115
116<h3>1.2 Generating text from the Markov chains</h3>
117
118<p>To generate the text, we will need a initializer word and the length of the output text to generate. Then, we perform a loop and get words based on the Markov chains probabilities calculated in the previous step.</p>
119
120<pre><code class="language-go">func (markov Markov) generateText(states []State, initWord string, count int) string {
121 var generatedText []string
122 word := initWord
123 generatedText = append(generatedText, word)
124 for i := 0; i < count; i++ {
125 word = getNextMarkovState(states, word)
126 if word == "word no exist on the memory" {
127 return "word no exist on the memory"
128 }
129 generatedText = append(generatedText, word)
130 }
131 text := strings.Join(generatedText, " ")
132 return text
133}
134</code></pre>
135
136<p>To generate the text we need a function that given the markov chains and a word, returns a random probability based word to be the next word following the given word:</p>
137
138<pre><code class="language-go">func getNextMarkovState(states []State, word string) string {
139 iState := -1
140 for i := 0; i < len(states); i++ {
141 if states[i].Word == word {
142 iState = i
143 }
144 }
145 if iState < 0 {
146 return "word no exist on the memory"
147 }
148 var next State
149 next = states[iState].NextStates[0]
150 next.Prob = rand.Float64() * states[iState].Prob
151 for i := 0; i < len(states[iState].NextStates); i++ {
152 if (rand.Float64()*states[iState].NextStates[i].Prob) > next.Prob && states[iState-1].Word != states[iState].NextStates[i].Word {
153 next = states[iState].NextStates[i]
154 }
155 }
156 return next.Word
157}
158</code></pre>
159
160<h2>2. Twitter API</h2>
161
162<p>To interact with the Twitter API, we will use <strong>go-twitter</strong> library <a href="https://github.com/dghubble/go-twitter">https://github.com/dghubble/go-twitter</a> .</p>
163
164<p>We setup a streaming connection with the Twitter API, we will filter tweets by some words related to our input dataset:</p>
165
166<pre><code class="language-go">func startStreaming(states []State, flock Flock, flockUser *twitter.Client, botScreenName string, keywords []string) {
167 // Convenience Demux demultiplexed stream messages
168 demux := twitter.NewSwitchDemux()
169 demux.Tweet = func(tweet *twitter.Tweet) {
170 if isRT(tweet) == false && isFromBot(flock, tweet) == false {
171 processTweet(states, flockUser, botScreenName, keywords, tweet)
172 }
173 }
174 demux.DM = func(dm *twitter.DirectMessage) {
175 fmt.Println(dm.SenderID)
176 }
177 demux.Event = func(event *twitter.Event) {
178 fmt.Printf("%#v\n", event)
179 }
180
181 fmt.Println("Starting Stream...")
182 // FILTER
183 filterParams := &twitter.StreamFilterParams{
184 Track: keywords,
185 StallWarnings: twitter.Bool(true),
186 }
187 stream, err := flockUser.Streams.Filter(filterParams)
188 if err != nil {
189 log.Fatal(err)
190 }
191 // Receive messages until stopped or stream quits
192 demux.HandleChan(stream.Messages)
193}
194</code></pre>
195
196<p>Then, each time that a new tweet with some of our tracking words is tweeted, we process that tweet and then we generate a reply based on the Markov chains, and we post that reply:</p>
197
198<pre><code class="language-go">func processTweet(states []State, flockUser *twitter.Client, botScreenName string, keywords []string, tweet *twitter.Tweet) {
199 c.Yellow("bot @" + botScreenName + " - New tweet detected:")
200 fmt.Println(tweet.Text)
201
202 tweetWords := strings.Split(tweet.Text, " ")
203 generatedText := "word no exist on the memory"
204 for i := 0; i < len(tweetWords) && generatedText == "word no exist on the memory"; i++ {
205 fmt.Println(strconv.Itoa(i) + " - " + tweetWords[i])
206 generatedText = generateMarkovResponse(states, tweetWords[i])
207 }
208 c.Yellow("bot @" + botScreenName + " posting response")
209 fmt.Println(tweet.ID)
210 replyTweet(flockUser, "@"+tweet.User.ScreenName+" "+generatedText, tweet.ID)
211 waitTime(1)
212}
213</code></pre>
214
215<pre><code class="language-go">func replyTweet(client *twitter.Client, text string, inReplyToStatusID int64) {
216 tweet, httpResp, err := client.Statuses.Update(text, &twitter.StatusUpdateParams{
217 InReplyToStatusID: inReplyToStatusID,
218 })
219 if err != nil {
220 fmt.Println(err)
221 }
222 if httpResp.Status != "200 OK" {
223 c.Red("error: " + httpResp.Status)
224 c.Purple("maybe twitter has blocked the account, CTRL+C, wait 15 minutes and try again")
225 }
226 fmt.Print("tweet posted: ")
227 c.Green(tweet.Text)
228}
229</code></pre>
230
231<h2>3. Flock-Botnet, or how to avoid the Twitter API limitations</h2>
232
233<p>If you ever played with the Twitter API, you will have seen that there are some restrictions and limitations. That means that if your bot have too much posting activity, the account will get blocked for some minutes.</p>
234
235<p>To avoid this limitation, we will deploy a botnet, where each bot will be replying tweets based on the Markov chains probabilities. In this way, when a bot post a tweet reply, the bot falls asleep for 1 minute. In the meantime, the other bots will be processing and replying the other tweets.</p>
236
237<p><img src="img/posts/flock-botnet/flock-botnet-scheme.png" alt="flock-botnet" title="01" /></p>
238
239<h2>3. Putting it all together</h2>
240
241<p>In this demo, we will use only 3 bots (twitter accounts).</p>
242
243<p>The botnet configuration will be in the config.json file:</p>
244
245<pre><code class="language-json">[{
246 "title": "bot1",
247 "consumer_key": "xxxxxxxxxxxxx",
248 "consumer_secret": "xxxxxxxxxxxxx",
249 "access_token_key": "xxxxxxxxxxxxx",
250 "access_token_secret": "xxxxxxxxxxxxx"
251 },
252 {
253 "title": "bot2",
254 "consumer_key": "xxxxxxxxxxxxx",
255 "consumer_secret": "xxxxxxxxxxxxx",
256 "access_token_key": "xxxxxxxxxxxxx",
257 "access_token_secret": "xxxxxxxxxxxxx"
258 },
259 {
260 "title": "bot3",
261 "consumer_key": "xxxxxxxxxxxxx",
262 "consumer_secret": "xxxxxxxxxxxxx",
263 "access_token_key": "xxxxxxxxxxxxx",
264 "access_token_secret": "xxxxxxxxxxxxx"
265 }
266]
267</code></pre>
268
269<p>The complete process will be:</p>
270
271<p><img src="img/posts/flock-botnet/steps.png" alt="flock-botnet" title="01" /></p>
272
273<h2>4. Demo</h2>
274
275<p>We have setted up a small demo with 3 bots. As we have said in the beging of this post, we have used the <a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">“The Critique of Pure Reason”, by Immanuel Kant </a> to generate the Markov chains.</p>
276
277<p>When the botnet is up working, the bots start streaming all the twitter new tweets containing the configured keywords.</p>
278
279<p>Each bot takes a tweet, analyzes the containing words, and generates a reply using the Markov chains previously calculated, and posts the tweet as reply.</p>
280
281<p>Example of terminal view during the flock-botnet execution:</p>
282
283<p><img src="img/posts/flock-botnet/terminal00.png" alt="flock-botnet" title="01" /></p>
284
285<p>Here is an example of the execution:</p>
286
287<p><img src="img/posts/flock-botnet/flock-botnet-demo.gif" alt="flock-botnet" title="Markov chain" /></p>
288
289<p>In the following examples, there are some screenshots that the bots (“@andreimarkov”, “@dodecahedron”, “@projectNSA”) have been replying to some people.</p>
290
291<p><img src="img/posts/flock-botnet/01.png" alt="flock-botnet" title="01" /></p>
292
293<hr>
294
295<p><img src="img/posts/flock-botnet/02.jpeg" alt="flock-botnet" title="02" /></p>
296
297<hr>
298
299<p><img src="img/posts/flock-botnet/03.jpeg" alt="flock-botnet" title="03" /></p>
300
301<hr>
302
303<p><img src="img/posts/flock-botnet/04.jpeg" alt="flock-botnet" title="04" /></p>
304
305<h2>Conclusion</h2>
306
307<p>In this article, we have seen how to build a Twitter botnet with the bots replying tweets with text generated based on Markov chains.</p>
308
309<p>As in this article we have used only 1 grade Markov chains, so the generated text is not really like humans text. But for future projects, a good choice would be combining more grades Markov chains with other text mining techniques.</p>
310
311<p>Twitter API have lots of uses, and in this post we have seen one of them. I hope to be able to write some more articles about other projects arround the Twitter API. For example some Twitter <a href="https://devpost.com/software/projectnsa">network nodes analysis</a>, or some <a href="https://arnaucube.com/hashtagsUsersNetworkPage.html">users & hashtags analysis</a>.</p>
312
313<p>The complete code of this project is able in <a href="https://github.com/arnaucube/flock-botnet">https://github.com/arnaucube/flock-botnet</a></p>
314
315<p>Project page: <a href="http://arnaucube.com/flock-botnet/">http://arnaucube.com/flock-botnet/</a></p>
316
317 </div>
318
319 <footer style="text-align:center; margin-top:100px;margin-bottom:50px;">
320 <div class="container">
321 <div class="row">
322 <ul class="list-inline">
323 <li><a href="https://twitter.com/arnaucube"
324 style="color:gray;text-decoration:none;"
325 target="_blank">twitter.com/arnaucube</a>
326 </li>
327 <li><a href="https://github.com/arnaucube"
328 style="color:gray;text-decoration:none;"
329 target="_blank">github.com/arnaucube</a>
330 </li>
331 </ul>
332 </div>
333 <div class="row" style="display:inline-block;">
334 Blog made with <a href="http://github.com/arnaucube/blogo/"
335 target="_blank" style="color: gray;text-decoration:none;">Blogo</a>
336 </div>
337 </div>
338 </footer>
339
340 <script>
341 </script>
342 <script src="js/external-links.js"></script>
343 <script>hljs.initHighlightingOnLoad();</script>
344 <script defer src="https://cdn.jsdelivr.net/npm/katex@0.13.11/dist/katex.min.js" integrity="sha384-YNHdsYkH6gMx9y3mRkmcJ2mFUjTd0qNQQvY9VYZgQd7DcN7env35GzlmFaZ23JGp" crossorigin="anonymous"></script>
345 <script defer src="https://cdn.jsdelivr.net/npm/katex@0.13.11/dist/contrib/auto-render.min.js" integrity="sha384-vZTG03m+2yp6N6BNi5iM4rW4oIwk5DfcNdFfxkk9ZWpDriOkXX8voJBFrAO7MpVl" crossorigin="anonymous"></script>
346 <script>
347 document.addEventListener("DOMContentLoaded", function() {
348 renderMathInElement(document.body, {
349 displayMode: false,
350 // customised options
351 // • auto-render specific keys, e.g.:
352 delimiters: [
353 {left: '$$', right: '$$', display: true},
354 {left: '$', right: '$', display: false},
355 ],
356 // • rendering keys, e.g.:
357 throwOnError : true
358 });
359 });
360</script>
361
362
363 </body>
364</html>
365