arnaucube.com/public/flock-botnet.html
public/flock-botnet.htmlRaw
1<!DOCTYPE html>
2<html lang="en">
3
4 <head>
5 <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
6 <meta charset="utf-8">
7 <title> Auto generated tweets from Markov chains - arnaucube - blog</title>
8 <meta name="title" content=" Auto generated tweets from Markov chains - arnaucube - blog">
9 <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
10
11 <meta property="og:title" content=" Auto generated tweets from Markov chains - arnaucube - blog" />
12 <meta property="og:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
13 <meta property="og:url" content="https://arnaucube.com/blog/flock-botnet.html" />
14 <meta property="og:type" content="article" />
15 <meta property="og:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png" />
16 <meta name="twitter:title" content=" Auto generated tweets from Markov chains - arnaucube - blog">
17 <meta name="twitter:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
18 <meta name="twitter:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png">
19 <meta name="twitter:card" content="summary_large_image">
20 <meta name="author" content="arnaucube">
21
22 <meta name="viewport" content="width=device-width, initial-scale=1">
23
24 <link href="css/bootstrap.min.css" rel="stylesheet">
25 <link rel="stylesheet" href="css/style.css">
26
27 <!-- highlightjs -->
28 <!-- <link rel="stylesheet" href="js/highlightjs/atom-one-dark.css"> -->
29 <link rel="stylesheet" href="js/highlightjs/gruvbox-dark.css">
30 <script src="js/highlightjs/highlight.pack.js"></script>
31
32 <!-- katex -->
33 <link rel="stylesheet" href="js/katex/katex.min.css">
34 </head>
35
36 <body>
37
38 <!-- o_gradient_background" -->
39 <nav id="mainNav" class="navbar navbar-default navbar-fixed-top"
40 style="height:50px;font-size:130%;">
41 <div class="container">
42 <a href="/blog" style="color:#000;">Blog index</a>
43 <div style="float:right;">
44 <a href="/" style="color:#000;display:inline-block;">arnaucube.com</a>
45 <div class="onoffswitch" style="margin:10px;display:inline-block;" title="change theme">
46 <input onclick="switchTheme()" type="checkbox" name="onoffswitch" class="onoffswitch-checkbox"
47 id="themeSwitcher">
48 <label class="onoffswitch-label" for="themeSwitcher"></label>
49 </div>
50 </div>
51 </div>
52 <img style="height:5px; width:100%; margin-top:8px;" src="img/gradient-line.jpg" />
53 </nav>
54
55
56
57 <div class="container" style="margin-top:40px;max-width:800px;">
58 <h1>Auto generated tweets from Markov chains</h1>
59
60<p><em>2017-12-29</em></p>
61
62<p>The main goal of this article is to explain how to develop a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains. As this is a project to learn, we have made everything from scratch.</p>
63
64<p>The idea of mixing twitter bots with Markov chains was in a Twitter conversation with <a href="https://twitter.com/x0rz">@x0rz</a></p>
65
66<h2>1. Markov chains</h2>
67
68<p>A Markov chain is a sequence of stochastic events (based on probabilities) where the current state of a variable or system is independent of all past states, except the current state.</p>
69
70<p><a href="https://en.wikipedia.org/wiki/Markov_chain">https://en.wikipedia.org/wiki/Markov_chain</a></p>
71
72<p>In our case, we will use Markov chains to analyze the probabilities that after a word comes another concrete word. So, we will generate some diagram like the following one, but with thousands of words.</p>
73
74<p><img src="img/posts/flock-botnet/markovchain.png" alt="flock-botnet" title="Markov chain" /></p>
75
76<p>In our case, we need as input some text document with thousands of words, to get a better input data. In this example we have made it with the book “The Critique of Pure Reason”, by Immanuel Kant (<a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">http://www.gutenberg.org/cache/epub/4280/pg4280.txt</a>), just because is the first book that we have found in .txt format.</p>
77
78<h3>1.1 Calculating the Markov chains</h3>
79
80<p>First we need to read the text file:</p>
81
82<pre><code class="language-go">func readTxt(path string) (string, error) {
83 data, err := ioutil.ReadFile(path)
84 if err != nil {
85 //Do something
86 }
87 dataClean := strings.Replace(string(data), "\n", " ", -1)
88 content := string(dataClean)
89 return content, err
90}
91</code></pre>
92
93<p>To calculate the probabilities of the Markov states, we have made the following function that analyzes the full input text, and stores the Markov states:</p>
94
95<pre><code class="language-go">func calcMarkovStates(words []string) []State {
96 var states []State
97 //count words
98 for i := 0; i < len(words)-1; i++ {
99 var iState int
100 states, iState = addWordToStates(states, words[i])
101 if iState < len(words) {
102 states[iState].NextStates, _ = addWordToStates(states[iState].NextStates, words[i+1])
103 }
104
105 printLoading(i, len(words))
106 }
107
108 //count prob
109 for i := 0; i < len(states); i++ {
110 states[i].Prob = (float64(states[i].Count) / float64(len(words)) * 100)
111 for j := 0; j < len(states[i].NextStates); j++ {
112 states[i].NextStates[j].Prob = (float64(states[i].NextStates[j].Count) / float64(len(words)) * 100)
113 }
114 }
115 fmt.Println("\ntotal words computed: " + strconv.Itoa(len(words)))
116 return states
117}
118</code></pre>
119
120<p>The <em>printLoading</em> function, is just a simple function to print in the terminal the % of the process done:</p>
121
122<pre><code class="language-go">func printLoading(n int, total int) {
123 var bar []string
124 tantPerFourty := int((float64(n) / float64(total)) * 40)
125 tantPerCent := int((float64(n) / float64(total)) * 100)
126 for i := 0; i < tantPerFourty; i++ {
127 bar = append(bar, "█")
128 }
129 progressBar := strings.Join(bar, "")
130 fmt.Printf("\r " + progressBar + " - " + strconv.Itoa(tantPerCent) + "")
131}
132</code></pre>
133
134<p><img src="img/posts/flock-botnet/progressbarMarkov.gif" alt="flock-botnet" title="Markov chain" /></p>
135
136<h3>1.2 Generating text from the Markov chains</h3>
137
138<p>To generate the text, we will need a initializer word and the length of the output text to generate. Then, we perform a loop and get words based on the Markov chains probabilities calculated in the previous step.</p>
139
140<pre><code class="language-go">func (markov Markov) generateText(states []State, initWord string, count int) string {
141 var generatedText []string
142 word := initWord
143 generatedText = append(generatedText, word)
144 for i := 0; i < count; i++ {
145 word = getNextMarkovState(states, word)
146 if word == "word no exist on the memory" {
147 return "word no exist on the memory"
148 }
149 generatedText = append(generatedText, word)
150 }
151 text := strings.Join(generatedText, " ")
152 return text
153}
154</code></pre>
155
156<p>To generate the text we need a function that given the markov chains and a word, returns a random probability based word to be the next word following the given word:</p>
157
158<pre><code class="language-go">func getNextMarkovState(states []State, word string) string {
159 iState := -1
160 for i := 0; i < len(states); i++ {
161 if states[i].Word == word {
162 iState = i
163 }
164 }
165 if iState < 0 {
166 return "word no exist on the memory"
167 }
168 var next State
169 next = states[iState].NextStates[0]
170 next.Prob = rand.Float64() * states[iState].Prob
171 for i := 0; i < len(states[iState].NextStates); i++ {
172 if (rand.Float64()*states[iState].NextStates[i].Prob) > next.Prob && states[iState-1].Word != states[iState].NextStates[i].Word {
173 next = states[iState].NextStates[i]
174 }
175 }
176 return next.Word
177}
178</code></pre>
179
180<h2>2. Twitter API</h2>
181
182<p>To interact with the Twitter API, we will use <strong>go-twitter</strong> library <a href="https://github.com/dghubble/go-twitter">https://github.com/dghubble/go-twitter</a> .</p>
183
184<p>We setup a streaming connection with the Twitter API, we will filter tweets by some words related to our input dataset:</p>
185
186<pre><code class="language-go">func startStreaming(states []State, flock Flock, flockUser *twitter.Client, botScreenName string, keywords []string) {
187 // Convenience Demux demultiplexed stream messages
188 demux := twitter.NewSwitchDemux()
189 demux.Tweet = func(tweet *twitter.Tweet) {
190 if isRT(tweet) == false && isFromBot(flock, tweet) == false {
191 processTweet(states, flockUser, botScreenName, keywords, tweet)
192 }
193 }
194 demux.DM = func(dm *twitter.DirectMessage) {
195 fmt.Println(dm.SenderID)
196 }
197 demux.Event = func(event *twitter.Event) {
198 fmt.Printf("%#v\n", event)
199 }
200
201 fmt.Println("Starting Stream...")
202 // FILTER
203 filterParams := &twitter.StreamFilterParams{
204 Track: keywords,
205 StallWarnings: twitter.Bool(true),
206 }
207 stream, err := flockUser.Streams.Filter(filterParams)
208 if err != nil {
209 log.Fatal(err)
210 }
211 // Receive messages until stopped or stream quits
212 demux.HandleChan(stream.Messages)
213}
214</code></pre>
215
216<p>Then, each time that a new tweet with some of our tracking words is tweeted, we process that tweet and then we generate a reply based on the Markov chains, and we post that reply:</p>
217
218<pre><code class="language-go">func processTweet(states []State, flockUser *twitter.Client, botScreenName string, keywords []string, tweet *twitter.Tweet) {
219 c.Yellow("bot @" + botScreenName + " - New tweet detected:")
220 fmt.Println(tweet.Text)
221
222 tweetWords := strings.Split(tweet.Text, " ")
223 generatedText := "word no exist on the memory"
224 for i := 0; i < len(tweetWords) && generatedText == "word no exist on the memory"; i++ {
225 fmt.Println(strconv.Itoa(i) + " - " + tweetWords[i])
226 generatedText = generateMarkovResponse(states, tweetWords[i])
227 }
228 c.Yellow("bot @" + botScreenName + " posting response")
229 fmt.Println(tweet.ID)
230 replyTweet(flockUser, "@"+tweet.User.ScreenName+" "+generatedText, tweet.ID)
231 waitTime(1)
232}
233</code></pre>
234
235<pre><code class="language-go">func replyTweet(client *twitter.Client, text string, inReplyToStatusID int64) {
236 tweet, httpResp, err := client.Statuses.Update(text, &twitter.StatusUpdateParams{
237 InReplyToStatusID: inReplyToStatusID,
238 })
239 if err != nil {
240 fmt.Println(err)
241 }
242 if httpResp.Status != "200 OK" {
243 c.Red("error: " + httpResp.Status)
244 c.Purple("maybe twitter has blocked the account, CTRL+C, wait 15 minutes and try again")
245 }
246 fmt.Print("tweet posted: ")
247 c.Green(tweet.Text)
248}
249</code></pre>
250
251<h2>3. Flock-Botnet, or how to avoid the Twitter API limitations</h2>
252
253<p>If you ever played with the Twitter API, you will have seen that there are some restrictions and limitations. That means that if your bot have too much posting activity, the account will get blocked for some minutes.</p>
254
255<p>To avoid this limitation, we will deploy a botnet, where each bot will be replying tweets based on the Markov chains probabilities. In this way, when a bot post a tweet reply, the bot falls asleep for 1 minute. In the meantime, the other bots will be processing and replying the other tweets.</p>
256
257<p><img src="img/posts/flock-botnet/flock-botnet-scheme.png" alt="flock-botnet" title="01" /></p>
258
259<h2>3. Putting it all together</h2>
260
261<p>In this demo, we will use only 3 bots (twitter accounts).</p>
262
263<p>The botnet configuration will be in the config.json file:</p>
264
265<pre><code class="language-json">[{
266 "title": "bot1",
267 "consumer_key": "xxxxxxxxxxxxx",
268 "consumer_secret": "xxxxxxxxxxxxx",
269 "access_token_key": "xxxxxxxxxxxxx",
270 "access_token_secret": "xxxxxxxxxxxxx"
271 },
272 {
273 "title": "bot2",
274 "consumer_key": "xxxxxxxxxxxxx",
275 "consumer_secret": "xxxxxxxxxxxxx",
276 "access_token_key": "xxxxxxxxxxxxx",
277 "access_token_secret": "xxxxxxxxxxxxx"
278 },
279 {
280 "title": "bot3",
281 "consumer_key": "xxxxxxxxxxxxx",
282 "consumer_secret": "xxxxxxxxxxxxx",
283 "access_token_key": "xxxxxxxxxxxxx",
284 "access_token_secret": "xxxxxxxxxxxxx"
285 }
286]
287</code></pre>
288
289<p>The complete process will be:</p>
290
291<p><img src="img/posts/flock-botnet/steps.png" alt="flock-botnet" title="01" /></p>
292
293<h2>4. Demo</h2>
294
295<p>We have setted up a small demo with 3 bots. As we have said in the beging of this post, we have used the <a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">“The Critique of Pure Reason”, by Immanuel Kant </a> to generate the Markov chains.</p>
296
297<p>When the botnet is up working, the bots start streaming all the twitter new tweets containing the configured keywords.</p>
298
299<p>Each bot takes a tweet, analyzes the containing words, and generates a reply using the Markov chains previously calculated, and posts the tweet as reply.</p>
300
301<p>Example of terminal view during the flock-botnet execution:</p>
302
303<p><img src="img/posts/flock-botnet/terminal00.png" alt="flock-botnet" title="01" /></p>
304
305<p>Here is an example of the execution:</p>
306
307<p><img src="img/posts/flock-botnet/flock-botnet-demo.gif" alt="flock-botnet" title="Markov chain" /></p>
308
309<p>In the following examples, there are some screenshots that the bots (“@andreimarkov”, “@dodecahedron”, “@projectNSA”) have been replying to some people.</p>
310
311<p><img src="img/posts/flock-botnet/01.png" alt="flock-botnet" title="01" /></p>
312
313<hr>
314
315<p><img src="img/posts/flock-botnet/02.jpeg" alt="flock-botnet" title="02" /></p>
316
317<hr>
318
319<p><img src="img/posts/flock-botnet/03.jpeg" alt="flock-botnet" title="03" /></p>
320
321<hr>
322
323<p><img src="img/posts/flock-botnet/04.jpeg" alt="flock-botnet" title="04" /></p>
324
325<h2>Conclusion</h2>
326
327<p>In this article, we have seen how to build a Twitter botnet with the bots replying tweets with text generated based on Markov chains.</p>
328
329<p>As in this article we have used only 1 grade Markov chains, so the generated text is not really like humans text. But for future projects, a good choice would be combining more grades Markov chains with other text mining techniques.</p>
330
331<p>Twitter API have lots of uses, and in this post we have seen one of them. I hope to be able to write some more articles about other projects arround the Twitter API. For example some Twitter <a href="https://devpost.com/software/projectnsa">network nodes analysis</a>, or some <a href="https://arnaucube.com/hashtagsUsersNetworkPage.html">users & hashtags analysis</a>.</p>
332
333<p>The complete code of this project is able in <a href="https://github.com/arnaucube/flock-botnet">https://github.com/arnaucube/flock-botnet</a></p>
334
335<p>Project page: <a href="http://arnaucube.com/flock-botnet/">http://arnaucube.com/flock-botnet/</a></p>
336
337 </div>
338
339 <footer style="text-align:center; margin-top:100px;margin-bottom:50px;">
340 <div class="container">
341 <div class="row">
342 <ul class="list-inline">
343 <li><a href="https://twitter.com/arnaucube"
344 style="color:gray;text-decoration:none;"
345 target="_blank">twitter.com/arnaucube</a>
346 </li>
347 <li><a href="https://github.com/arnaucube"
348 style="color:gray;text-decoration:none;"
349 target="_blank">github.com/arnaucube</a>
350 </li>
351 </ul>
352 </div>
353 <div class="row" style="display:inline-block;">
354 Blog made with <a href="http://github.com/arnaucube/blogo/"
355 target="_blank" style="color: gray;text-decoration:none;">Blogo</a>
356 </div>
357 </div>
358 </footer>
359
360 <script>
361 </script>
362 <script src="js/external-links.js"></script>
363 <script>hljs.initHighlightingOnLoad();</script>
364 <script defer src="js/katex/katex.min.js"></script>
365 <script defer src="js/katex/auto-render.min.js"></script>
366 <script>
367 document.addEventListener("DOMContentLoaded", function() {
368 renderMathInElement(document.body, {
369 displayMode: false,
370 // customised options
371 // • auto-render specific keys, e.g.:
372 delimiters: [
373 {left: '$$', right: '$$', display: true},
374 {left: '$', right: '$', display: false},
375 {left: "\\[", right: "\\]", display: true},
376 {left: "\\(", right: "\\)", display: false},
377 ],
378 // • rendering keys, e.g.:
379 throwOnError : true
380 });
381 });
382
383 ///
384 let theme = localStorage.getItem("theme");
385 if ((theme === "light-theme")||(theme==null)) {
386 theme = "light-theme";
387 document.getElementById("themeSwitcher").checked = false;
388 } else if (theme === "dark-theme") {
389 theme = "dark-theme";
390 document.getElementById("themeSwitcher").checked = true;
391 }
392 document.body.className = theme;
393 localStorage.setItem("theme", theme);
394
395 function switchTheme() {
396 theme = localStorage.getItem("theme");
397 if (theme === "light-theme") {
398 theme = "dark-theme";
399 document.getElementById("themeSwitcher").checked = true;
400 } else {
401 theme = "light-theme";
402 document.getElementById("themeSwitcher").checked = false;
403 }
404 document.body.className = theme;
405 localStorage.setItem("theme", theme);
406
407 console.log(theme);
408 }
409 </script>
410 <script>
411 function tagLinks(tagName) {
412 var tags = document.getElementsByTagName(tagName);
413 for (var i=0, hElem; hElem = tags[i]; i++) {
414 if (hElem.parentNode.className=="row postThumb") {
415 continue;
416 }
417 hElem.id = hElem.innerHTML.toLowerCase().replace(" ", "-");
418 hElem.innerHTML = "<a style='text-decoration:none;color:black;' href='#"+hElem.id+"'>"+hElem.innerHTML+"</a>";
419 }
420 }
421 tagLinks("h2");
422 tagLinks("h3");
423 tagLinks("h4");
424 tagLinks("h5");
425 </script>
426 <script src="js/mermaid.min.js"></script>
427
428
429 </body>
430</html>
431