arnaucube.com/blog/flock-botnet.html
blog/flock-botnet.htmlRaw
1<!DOCTYPE html>
2<html lang="en">
3
4 <head>
5 <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
6 <meta charset="utf-8">
7 <title> Auto generated tweets from Markov chains - arnaucube - blog</title>
8 <meta name="title" content=" Auto generated tweets from Markov chains - arnaucube - blog">
9 <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
10
11 <meta property="og:title" content=" Auto generated tweets from Markov chains - arnaucube - blog" />
12 <meta property="og:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
13 <meta property="og:url" content="https://arnaucube.com/blog/flock-botnet.html" />
14 <meta property="og:type" content="article" />
15 <meta property="og:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png" />
16 <meta name="twitter:title" content=" Auto generated tweets from Markov chains - arnaucube - blog">
17 <meta name="twitter:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
18 <meta name="twitter:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png">
19 <meta name="twitter:card" content="summary_large_image">
20 <meta name="author" content="arnaucube">
21
22 <link rel="icon" type="image/png" href="img/logoArnauCubeFavicon.png">
23
24 <meta name="viewport" content="width=device-width, initial-scale=1">
25
26 <link href="css/bootstrap.min.css" rel="stylesheet">
27 <link rel="stylesheet" href="css/style.css">
28
29 <!-- highlightjs -->
30 <!-- <link rel="stylesheet" href="js/highlightjs/atom-one-dark.css"> -->
31 <link rel="stylesheet" href="js/highlightjs/atom-one-light.css">
32 <!-- <link rel="stylesheet" href="js/highlightjs/gruvbox-dark.css"> -->
33 <script src="js/highlightjs/highlight.pack.js"></script>
34
35 <!-- katex -->
36 <link rel="stylesheet" href="js/katex/katex.min.css">
37 </head>
38
39 <body>
40
41 <!-- o_gradient_background" -->
42 <nav id="mainNav" class="navbar navbar-default navbar-fixed-top"
43 style="height:50px;font-size:100%;">
44 <div class="container">
45 <div style="float:left;">
46 <a href="/blog" style="color:#000;display:inline-block;">blog</a>
47 <span style="margin-right:20px; margin-left:20px;">|</span>
48 <a href="/blog/notes.html" style="font-size:90%;color:#000;display:inline-block;">notes</a>
49 </div>
50 <div style="float:right;">
51 <a href="/" style="color:#000;display:inline-block;margin-right:10px;">arnaucube.com</a>
52 <div class="onoffswitch" style="margin-left:0px;display:inline-block;" title="change theme">
53 <input onclick="switchTheme()" type="checkbox" name="onoffswitch" class="onoffswitch-checkbox"
54 id="themeSwitcher">
55 <label class="onoffswitch-label" for="themeSwitcher"></label>
56 </div>
57 </div>
58 </div>
59 <!-- <img style="height:5px; width:100%; margin-top:8px;" src="img/gradient-line.jpg" /> -->
60 </nav>
61
62
63
64 <div class="container" style="margin-top:0px;max-width:800px;">
65 <h1>Auto generated tweets from Markov chains</h1>
66
67<p><em>2017-12-29</em></p>
68
69<p>The main goal of this article is to explain how to develop a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains. As this is a project to learn, we have made everything from scratch.</p>
70
71<p>The idea of mixing twitter bots with Markov chains was in a Twitter conversation with <a href="https://twitter.com/x0rz">@x0rz</a></p>
72
73<h2>1. Markov chains</h2>
74
75<p>A Markov chain is a sequence of stochastic events (based on probabilities) where the current state of a variable or system is independent of all past states, except the current state.</p>
76
77<p><a href="https://en.wikipedia.org/wiki/Markov_chain">https://en.wikipedia.org/wiki/Markov_chain</a></p>
78
79<p>In our case, we will use Markov chains to analyze the probabilities that after a word comes another concrete word. So, we will generate some diagram like the following one, but with thousands of words.</p>
80
81<p><img src="img/posts/flock-botnet/markovchain.png" alt="flock-botnet" title="Markov chain" /></p>
82
83<p>In our case, we need as input some text document with thousands of words, to get a better input data. In this example we have made it with the book “The Critique of Pure Reason”, by Immanuel Kant (<a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">http://www.gutenberg.org/cache/epub/4280/pg4280.txt</a>), just because is the first book that we have found in .txt format.</p>
84
85<h3>1.1 Calculating the Markov chains</h3>
86
87<p>First we need to read the text file:</p>
88
89<pre><code class="language-go">func readTxt(path string) (string, error) {
90 data, err := ioutil.ReadFile(path)
91 if err != nil {
92 //Do something
93 }
94 dataClean := strings.Replace(string(data), "\n", " ", -1)
95 content := string(dataClean)
96 return content, err
97}
98</code></pre>
99
100<p>To calculate the probabilities of the Markov states, we have made the following function that analyzes the full input text, and stores the Markov states:</p>
101
102<pre><code class="language-go">func calcMarkovStates(words []string) []State {
103 var states []State
104 //count words
105 for i := 0; i < len(words)-1; i++ {
106 var iState int
107 states, iState = addWordToStates(states, words[i])
108 if iState < len(words) {
109 states[iState].NextStates, _ = addWordToStates(states[iState].NextStates, words[i+1])
110 }
111
112 printLoading(i, len(words))
113 }
114
115 //count prob
116 for i := 0; i < len(states); i++ {
117 states[i].Prob = (float64(states[i].Count) / float64(len(words)) * 100)
118 for j := 0; j < len(states[i].NextStates); j++ {
119 states[i].NextStates[j].Prob = (float64(states[i].NextStates[j].Count) / float64(len(words)) * 100)
120 }
121 }
122 fmt.Println("\ntotal words computed: " + strconv.Itoa(len(words)))
123 return states
124}
125</code></pre>
126
127<p>The <em>printLoading</em> function, is just a simple function to print in the terminal the % of the process done:</p>
128
129<pre><code class="language-go">func printLoading(n int, total int) {
130 var bar []string
131 tantPerFourty := int((float64(n) / float64(total)) * 40)
132 tantPerCent := int((float64(n) / float64(total)) * 100)
133 for i := 0; i < tantPerFourty; i++ {
134 bar = append(bar, "█")
135 }
136 progressBar := strings.Join(bar, "")
137 fmt.Printf("\r " + progressBar + " - " + strconv.Itoa(tantPerCent) + "")
138}
139</code></pre>
140
141<p><img src="img/posts/flock-botnet/progressbarMarkov.gif" alt="flock-botnet" title="Markov chain" /></p>
142
143<h3>1.2 Generating text from the Markov chains</h3>
144
145<p>To generate the text, we will need a initializer word and the length of the output text to generate. Then, we perform a loop and get words based on the Markov chains probabilities calculated in the previous step.</p>
146
147<pre><code class="language-go">func (markov Markov) generateText(states []State, initWord string, count int) string {
148 var generatedText []string
149 word := initWord
150 generatedText = append(generatedText, word)
151 for i := 0; i < count; i++ {
152 word = getNextMarkovState(states, word)
153 if word == "word no exist on the memory" {
154 return "word no exist on the memory"
155 }
156 generatedText = append(generatedText, word)
157 }
158 text := strings.Join(generatedText, " ")
159 return text
160}
161</code></pre>
162
163<p>To generate the text we need a function that given the markov chains and a word, returns a random probability based word to be the next word following the given word:</p>
164
165<pre><code class="language-go">func getNextMarkovState(states []State, word string) string {
166 iState := -1
167 for i := 0; i < len(states); i++ {
168 if states[i].Word == word {
169 iState = i
170 }
171 }
172 if iState < 0 {
173 return "word no exist on the memory"
174 }
175 var next State
176 next = states[iState].NextStates[0]
177 next.Prob = rand.Float64() * states[iState].Prob
178 for i := 0; i < len(states[iState].NextStates); i++ {
179 if (rand.Float64()*states[iState].NextStates[i].Prob) > next.Prob && states[iState-1].Word != states[iState].NextStates[i].Word {
180 next = states[iState].NextStates[i]
181 }
182 }
183 return next.Word
184}
185</code></pre>
186
187<h2>2. Twitter API</h2>
188
189<p>To interact with the Twitter API, we will use <strong>go-twitter</strong> library <a href="https://github.com/dghubble/go-twitter">https://github.com/dghubble/go-twitter</a> .</p>
190
191<p>We setup a streaming connection with the Twitter API, we will filter tweets by some words related to our input dataset:</p>
192
193<pre><code class="language-go">func startStreaming(states []State, flock Flock, flockUser *twitter.Client, botScreenName string, keywords []string) {
194 // Convenience Demux demultiplexed stream messages
195 demux := twitter.NewSwitchDemux()
196 demux.Tweet = func(tweet *twitter.Tweet) {
197 if isRT(tweet) == false && isFromBot(flock, tweet) == false {
198 processTweet(states, flockUser, botScreenName, keywords, tweet)
199 }
200 }
201 demux.DM = func(dm *twitter.DirectMessage) {
202 fmt.Println(dm.SenderID)
203 }
204 demux.Event = func(event *twitter.Event) {
205 fmt.Printf("%#v\n", event)
206 }
207
208 fmt.Println("Starting Stream...")
209 // FILTER
210 filterParams := &twitter.StreamFilterParams{
211 Track: keywords,
212 StallWarnings: twitter.Bool(true),
213 }
214 stream, err := flockUser.Streams.Filter(filterParams)
215 if err != nil {
216 log.Fatal(err)
217 }
218 // Receive messages until stopped or stream quits
219 demux.HandleChan(stream.Messages)
220}
221</code></pre>
222
223<p>Then, each time that a new tweet with some of our tracking words is tweeted, we process that tweet and then we generate a reply based on the Markov chains, and we post that reply:</p>
224
225<pre><code class="language-go">func processTweet(states []State, flockUser *twitter.Client, botScreenName string, keywords []string, tweet *twitter.Tweet) {
226 c.Yellow("bot @" + botScreenName + " - New tweet detected:")
227 fmt.Println(tweet.Text)
228
229 tweetWords := strings.Split(tweet.Text, " ")
230 generatedText := "word no exist on the memory"
231 for i := 0; i < len(tweetWords) && generatedText == "word no exist on the memory"; i++ {
232 fmt.Println(strconv.Itoa(i) + " - " + tweetWords[i])
233 generatedText = generateMarkovResponse(states, tweetWords[i])
234 }
235 c.Yellow("bot @" + botScreenName + " posting response")
236 fmt.Println(tweet.ID)
237 replyTweet(flockUser, "@"+tweet.User.ScreenName+" "+generatedText, tweet.ID)
238 waitTime(1)
239}
240</code></pre>
241
242<pre><code class="language-go">func replyTweet(client *twitter.Client, text string, inReplyToStatusID int64) {
243 tweet, httpResp, err := client.Statuses.Update(text, &twitter.StatusUpdateParams{
244 InReplyToStatusID: inReplyToStatusID,
245 })
246 if err != nil {
247 fmt.Println(err)
248 }
249 if httpResp.Status != "200 OK" {
250 c.Red("error: " + httpResp.Status)
251 c.Purple("maybe twitter has blocked the account, CTRL+C, wait 15 minutes and try again")
252 }
253 fmt.Print("tweet posted: ")
254 c.Green(tweet.Text)
255}
256</code></pre>
257
258<h2>3. Flock-Botnet, or how to avoid the Twitter API limitations</h2>
259
260<p>If you ever played with the Twitter API, you will have seen that there are some restrictions and limitations. That means that if your bot have too much posting activity, the account will get blocked for some minutes.</p>
261
262<p>To avoid this limitation, we will deploy a botnet, where each bot will be replying tweets based on the Markov chains probabilities. In this way, when a bot post a tweet reply, the bot falls asleep for 1 minute. In the meantime, the other bots will be processing and replying the other tweets.</p>
263
264<p><img src="img/posts/flock-botnet/flock-botnet-scheme.png" alt="flock-botnet" title="01" /></p>
265
266<h2>3. Putting it all together</h2>
267
268<p>In this demo, we will use only 3 bots (twitter accounts).</p>
269
270<p>The botnet configuration will be in the config.json file:</p>
271
272<pre><code class="language-json">[{
273 "title": "bot1",
274 "consumer_key": "xxxxxxxxxxxxx",
275 "consumer_secret": "xxxxxxxxxxxxx",
276 "access_token_key": "xxxxxxxxxxxxx",
277 "access_token_secret": "xxxxxxxxxxxxx"
278 },
279 {
280 "title": "bot2",
281 "consumer_key": "xxxxxxxxxxxxx",
282 "consumer_secret": "xxxxxxxxxxxxx",
283 "access_token_key": "xxxxxxxxxxxxx",
284 "access_token_secret": "xxxxxxxxxxxxx"
285 },
286 {
287 "title": "bot3",
288 "consumer_key": "xxxxxxxxxxxxx",
289 "consumer_secret": "xxxxxxxxxxxxx",
290 "access_token_key": "xxxxxxxxxxxxx",
291 "access_token_secret": "xxxxxxxxxxxxx"
292 }
293]
294</code></pre>
295
296<p>The complete process will be:</p>
297
298<p><img src="img/posts/flock-botnet/steps.png" alt="flock-botnet" title="01" /></p>
299
300<h2>4. Demo</h2>
301
302<p>We have setted up a small demo with 3 bots. As we have said in the beging of this post, we have used the <a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">“The Critique of Pure Reason”, by Immanuel Kant </a> to generate the Markov chains.</p>
303
304<p>When the botnet is up working, the bots start streaming all the twitter new tweets containing the configured keywords.</p>
305
306<p>Each bot takes a tweet, analyzes the containing words, and generates a reply using the Markov chains previously calculated, and posts the tweet as reply.</p>
307
308<p>Example of terminal view during the flock-botnet execution:</p>
309
310<p><img src="img/posts/flock-botnet/terminal00.png" alt="flock-botnet" title="01" /></p>
311
312<p>Here is an example of the execution:</p>
313
314<p><img src="img/posts/flock-botnet/flock-botnet-demo.gif" alt="flock-botnet" title="Markov chain" /></p>
315
316<p>In the following examples, there are some screenshots that the bots (“@andreimarkov”, “@dodecahedron”, “@projectNSA”) have been replying to some people.</p>
317
318<p><img src="img/posts/flock-botnet/01.png" alt="flock-botnet" title="01" /></p>
319
320<hr>
321
322<p><img src="img/posts/flock-botnet/02.jpeg" alt="flock-botnet" title="02" /></p>
323
324<hr>
325
326<p><img src="img/posts/flock-botnet/03.jpeg" alt="flock-botnet" title="03" /></p>
327
328<hr>
329
330<p><img src="img/posts/flock-botnet/04.jpeg" alt="flock-botnet" title="04" /></p>
331
332<h2>Conclusion</h2>
333
334<p>In this article, we have seen how to build a Twitter botnet with the bots replying tweets with text generated based on Markov chains.</p>
335
336<p>As in this article we have used only 1 grade Markov chains, so the generated text is not really like humans text. But for future projects, a good choice would be combining more grades Markov chains with other text mining techniques.</p>
337
338<p>Twitter API have lots of uses, and in this post we have seen one of them. I hope to be able to write some more articles about other projects arround the Twitter API. For example some Twitter <a href="https://devpost.com/software/projectnsa">network nodes analysis</a>, or some <a href="https://arnaucube.com/hashtagsUsersNetworkPage.html">users & hashtags analysis</a>.</p>
339
340<p>The complete code of this project is able in <a href="https://github.com/arnaucube/flock-botnet">https://github.com/arnaucube/flock-botnet</a></p>
341
342<p>Project page: <a href="http://arnaucube.com/flock-botnet/">http://arnaucube.com/flock-botnet/</a></p>
343
344 </div>
345
346 <footer style="text-align:center; margin-top:100px;margin-bottom:50px;">
347 <div class="container">
348 <br>
349 <a href="/blog">Go to main</a>
350 <br><br>
351 <div class="row">
352 <ul class="list-inline">
353 <li><a href="https://twitter.com/arnaucube"
354 style="color:gray;text-decoration:none;"
355 target="_blank">twitter.com/arnaucube</a>
356 </li>
357 <li><a href="https://github.com/arnaucube"
358 style="color:gray;text-decoration:none;"
359 target="_blank">github.com/arnaucube</a>
360 </li>
361 </ul>
362 </div>
363 <div class="row" style="display:inline-block;">
364 Blog made with <a href="http://github.com/arnaucube/blogo/"
365 target="_blank" style="color: gray;text-decoration:none;">Blogo</a>
366 </div>
367 </div>
368 </footer>
369
370 <script>
371 </script>
372 <script src="js/external-links.js"></script>
373 <script>hljs.initHighlightingOnLoad();</script>
374 <script defer src="js/katex/katex.min.js"></script>
375 <script defer src="js/katex/auto-render.min.js"></script>
376 <script>
377 document.addEventListener("DOMContentLoaded", function() {
378 renderMathInElement(document.body, {
379 displayMode: false,
380 // customised options
381 // • auto-render specific keys, e.g.:
382 delimiters: [
383 {left: '$$', right: '$$', display: true},
384 {left: '$', right: '$', display: false},
385 {left: "\\[", right: "\\]", display: true},
386 {left: "\\(", right: "\\)", display: false},
387 {left: "\\begin{equation}", right: "\\end{equation}", display: true}
388 ],
389 // • rendering keys, e.g.:
390 throwOnError : true
391 });
392 });
393
394 ///
395 let theme = localStorage.getItem("theme");
396 if ((theme === "light-theme")||(theme==null)) {
397 theme = "light-theme";
398 document.getElementById("themeSwitcher").checked = false;
399 } else if (theme === "dark-theme") {
400 theme = "dark-theme";
401 document.getElementById("themeSwitcher").checked = true;
402 }
403 document.body.className = theme;
404 localStorage.setItem("theme", theme);
405
406 function switchTheme() {
407 theme = localStorage.getItem("theme");
408 if (theme === "light-theme") {
409 theme = "dark-theme";
410 document.getElementById("themeSwitcher").checked = true;
411 } else {
412 theme = "light-theme";
413 document.getElementById("themeSwitcher").checked = false;
414 }
415 document.body.className = theme;
416 localStorage.setItem("theme", theme);
417
418 console.log(theme);
419 }
420 </script>
421 <script>
422 function tagLinks(tagName) {
423 var tags = document.getElementsByTagName(tagName);
424 for (var i=0, hElem; hElem = tags[i]; i++) {
425 if (hElem.parentNode.className=="row postThumb") {
426 continue;
427 }
428 hElem.id = hElem.innerHTML.toLowerCase().replace(" ", "-");
429 hElem.innerHTML = "<a style='text-decoration:none;color:black;' href='#"+hElem.id+"'>"+hElem.innerHTML+"</a>";
430 }
431 }
432 tagLinks("h2");
433 tagLinks("h3");
434 tagLinks("h4");
435 tagLinks("h5");
436 </script>
437 <script src="js/mermaid.min.js"></script>
438
439
440 </body>
441</html>
442