arnaucube
arnaucube/arnaucube.comPublic Self-hosted repository
arnaucube.com/blog/flock-botnet.html
blog/flock-botnet.htmlRaw
  1<!DOCTYPE html>
  2<html lang="en">
  3
  4  <head>
  5    <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
  6    <meta charset="utf-8">
  7    <title> Auto generated tweets from Markov chains - arnaucube - blog</title>
  8    <meta name="title" content=" Auto generated tweets from Markov chains - arnaucube - blog">
  9    <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
 10
 11    <meta property="og:title" content=" Auto generated tweets from Markov chains - arnaucube - blog" />
 12    <meta property="og:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
 13    <meta property="og:url" content="https://arnaucube.com/blog/flock-botnet.html" />
 14    <meta property="og:type" content="article" />
 15    <meta property="og:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png" />
 16    <meta name="twitter:title" content=" Auto generated tweets from Markov chains - arnaucube - blog">
 17    <meta name="twitter:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
 18    <meta name="twitter:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png">
 19    <meta name="twitter:card" content="summary_large_image">
 20    <meta name="author" content="arnaucube">
 21
 22    <link rel="icon" type="image/png" href="img/logoArnauCubeFavicon.png">
 23
 24    <meta name="viewport" content="width=device-width, initial-scale=1">
 25
 26    <link href="css/bootstrap.min.css" rel="stylesheet">
 27    <link rel="stylesheet" href="css/style.css">
 28
 29    <!-- highlightjs -->
 30    <!-- <link rel="stylesheet" href="js/highlightjs/atom-one-dark.css"> -->
 31    <link rel="stylesheet" href="js/highlightjs/atom-one-light.css">
 32    <!-- <link rel="stylesheet" href="js/highlightjs/gruvbox-dark.css"> -->
 33    <script src="js/highlightjs/highlight.pack.js"></script>
 34
 35    <!-- katex -->
 36    <link rel="stylesheet" href="js/katex/katex.min.css">
 37  </head>
 38
 39  <body>
 40
 41    <!-- o_gradient_background" -->
 42    <nav id="mainNav" class="navbar navbar-default navbar-fixed-top"
 43                      style="height:50px;font-size:100%;">
 44      <div class="container">
 45        <div style="float:left;">
 46        <a href="/blog" style="color:#000;display:inline-block;">blog</a>
 47        <span style="margin-right:20px; margin-left:20px;">|</span>
 48        <a href="/blog/notes.html" style="font-size:90%;color:#000;display:inline-block;">notes</a>
 49        </div>
 50        <div style="float:right;">
 51          <a href="/" style="color:#000;display:inline-block;margin-right:10px;">arnaucube.com</a>
 52          <div class="onoffswitch" style="margin-left:0px;display:inline-block;" title="change theme">
 53            <input onclick="switchTheme()" type="checkbox" name="onoffswitch" class="onoffswitch-checkbox"
 54              id="themeSwitcher">
 55            <label class="onoffswitch-label" for="themeSwitcher"></label>
 56          </div>
 57        </div>
 58      </div>
 59      <!-- <img style="height:5px; width:100%; margin-top:8px;" src="img/gradient-line.jpg" /> -->
 60    </nav>
 61
 62
 63
 64    <div class="container" style="margin-top:0px;max-width:800px;">
 65      <h1>Auto generated tweets from Markov chains</h1>
 66
 67<p><em>2017-12-29</em></p>
 68
 69<p>The main goal of this article is to explain how to develop a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains. As this is a project to learn, we have made everything from scratch.</p>
 70
 71<p>The idea of mixing twitter bots with Markov chains was in a Twitter conversation with <a href="https://twitter.com/x0rz">@x0rz</a></p>
 72
 73<h2>1. Markov chains</h2>
 74
 75<p>A Markov chain is a sequence of stochastic events (based on probabilities) where the current state of a variable or system is independent of all past states, except the current state.</p>
 76
 77<p><a href="https://en.wikipedia.org/wiki/Markov_chain">https://en.wikipedia.org/wiki/Markov_chain</a></p>
 78
 79<p>In our case, we will use Markov chains to analyze the probabilities that after a word comes another concrete word. So, we will generate some diagram like the following one, but with thousands of words.</p>
 80
 81<p><img src="img/posts/flock-botnet/markovchain.png" alt="flock-botnet" title="Markov chain" /></p>
 82
 83<p>In our case, we need as input some text document with thousands of words, to get a better input data. In this example we have made it with the book &ldquo;The Critique of Pure Reason&rdquo;, by Immanuel Kant (<a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">http://www.gutenberg.org/cache/epub/4280/pg4280.txt</a>), just because is the first book that we have found in .txt format.</p>
 84
 85<h3>1.1 Calculating the Markov chains</h3>
 86
 87<p>First we need to read the text file:</p>
 88
 89<pre><code class="language-go">func readTxt(path string) (string, error) {
 90	data, err := ioutil.ReadFile(path)
 91	if err != nil {
 92		//Do something
 93	}
 94	dataClean := strings.Replace(string(data), &quot;\n&quot;, &quot; &quot;, -1)
 95	content := string(dataClean)
 96	return content, err
 97}
 98</code></pre>
 99
100<p>To calculate the probabilities of the Markov states, we have made the following function that analyzes the full input text, and stores the Markov states:</p>
101
102<pre><code class="language-go">func calcMarkovStates(words []string) []State {
103	var states []State
104	//count words
105	for i := 0; i &lt; len(words)-1; i++ {
106		var iState int
107		states, iState = addWordToStates(states, words[i])
108		if iState &lt; len(words) {
109			states[iState].NextStates, _ = addWordToStates(states[iState].NextStates, words[i+1])
110		}
111
112		printLoading(i, len(words))
113	}
114
115	//count prob
116	for i := 0; i &lt; len(states); i++ {
117		states[i].Prob = (float64(states[i].Count) / float64(len(words)) * 100)
118		for j := 0; j &lt; len(states[i].NextStates); j++ {
119			states[i].NextStates[j].Prob = (float64(states[i].NextStates[j].Count) / float64(len(words)) * 100)
120		}
121	}
122	fmt.Println(&quot;\ntotal words computed: &quot; + strconv.Itoa(len(words)))
123	return states
124}
125</code></pre>
126
127<p>The <em>printLoading</em> function, is just a simple function to print in the terminal the % of the process done:</p>
128
129<pre><code class="language-go">func printLoading(n int, total int) {
130	var bar []string
131	tantPerFourty := int((float64(n) / float64(total)) * 40)
132	tantPerCent := int((float64(n) / float64(total)) * 100)
133	for i := 0; i &lt; tantPerFourty; i++ {
134		bar = append(bar, &quot;█&quot;)
135	}
136	progressBar := strings.Join(bar, &quot;&quot;)
137	fmt.Printf(&quot;\r &quot; + progressBar + &quot; - &quot; + strconv.Itoa(tantPerCent) + &quot;&quot;)
138}
139</code></pre>
140
141<p><img src="img/posts/flock-botnet/progressbarMarkov.gif" alt="flock-botnet" title="Markov chain" /></p>
142
143<h3>1.2 Generating text from the Markov chains</h3>
144
145<p>To generate the text, we will need a initializer word and the length of the output text to generate. Then, we perform a loop and get words based on the Markov chains probabilities calculated in the previous step.</p>
146
147<pre><code class="language-go">func (markov Markov) generateText(states []State, initWord string, count int) string {
148	var generatedText []string
149	word := initWord
150	generatedText = append(generatedText, word)
151	for i := 0; i &lt; count; i++ {
152		word = getNextMarkovState(states, word)
153		if word == &quot;word no exist on the memory&quot; {
154			return &quot;word no exist on the memory&quot;
155		}
156		generatedText = append(generatedText, word)
157	}
158	text := strings.Join(generatedText, &quot; &quot;)
159	return text
160}
161</code></pre>
162
163<p>To generate the text we need a function that given the markov chains and a word, returns a random probability based word to be the next word following the given word:</p>
164
165<pre><code class="language-go">func getNextMarkovState(states []State, word string) string {
166	iState := -1
167	for i := 0; i &lt; len(states); i++ {
168		if states[i].Word == word {
169			iState = i
170		}
171	}
172	if iState &lt; 0 {
173		return &quot;word no exist on the memory&quot;
174	}
175	var next State
176	next = states[iState].NextStates[0]
177	next.Prob = rand.Float64() * states[iState].Prob
178	for i := 0; i &lt; len(states[iState].NextStates); i++ {
179		if (rand.Float64()*states[iState].NextStates[i].Prob) &gt; next.Prob &amp;&amp; states[iState-1].Word != states[iState].NextStates[i].Word {
180			next = states[iState].NextStates[i]
181		}
182	}
183	return next.Word
184}
185</code></pre>
186
187<h2>2. Twitter API</h2>
188
189<p>To interact with the Twitter API, we will use <strong>go-twitter</strong> library <a href="https://github.com/dghubble/go-twitter">https://github.com/dghubble/go-twitter</a> .</p>
190
191<p>We setup a streaming connection with the Twitter API, we will filter tweets by some words related to our input dataset:</p>
192
193<pre><code class="language-go">func startStreaming(states []State, flock Flock, flockUser *twitter.Client, botScreenName string, keywords []string) {
194	// Convenience Demux demultiplexed stream messages
195	demux := twitter.NewSwitchDemux()
196	demux.Tweet = func(tweet *twitter.Tweet) {
197		if isRT(tweet) == false &amp;&amp; isFromBot(flock, tweet) == false {
198			processTweet(states, flockUser, botScreenName, keywords, tweet)
199		}
200	}
201	demux.DM = func(dm *twitter.DirectMessage) {
202		fmt.Println(dm.SenderID)
203	}
204	demux.Event = func(event *twitter.Event) {
205		fmt.Printf(&quot;%#v\n&quot;, event)
206	}
207
208	fmt.Println(&quot;Starting Stream...&quot;)
209	// FILTER
210	filterParams := &amp;twitter.StreamFilterParams{
211		Track:         keywords,
212		StallWarnings: twitter.Bool(true),
213	}
214	stream, err := flockUser.Streams.Filter(filterParams)
215	if err != nil {
216		log.Fatal(err)
217	}
218	// Receive messages until stopped or stream quits
219	demux.HandleChan(stream.Messages)
220}
221</code></pre>
222
223<p>Then, each time that a new tweet with some of our tracking words is tweeted, we process that tweet and then we generate a reply based on the Markov chains, and we post that reply:</p>
224
225<pre><code class="language-go">func processTweet(states []State, flockUser *twitter.Client, botScreenName string, keywords []string, tweet *twitter.Tweet) {
226	c.Yellow(&quot;bot @&quot; + botScreenName + &quot; - New tweet detected:&quot;)
227	fmt.Println(tweet.Text)
228
229	tweetWords := strings.Split(tweet.Text, &quot; &quot;)
230	generatedText := &quot;word no exist on the memory&quot;
231	for i := 0; i &lt; len(tweetWords) &amp;&amp; generatedText == &quot;word no exist on the memory&quot;; i++ {
232		fmt.Println(strconv.Itoa(i) + &quot; - &quot; + tweetWords[i])
233		generatedText = generateMarkovResponse(states, tweetWords[i])
234	}
235	c.Yellow(&quot;bot @&quot; + botScreenName + &quot; posting response&quot;)
236	fmt.Println(tweet.ID)
237	replyTweet(flockUser, &quot;@&quot;+tweet.User.ScreenName+&quot; &quot;+generatedText, tweet.ID)
238	waitTime(1)
239}
240</code></pre>
241
242<pre><code class="language-go">func replyTweet(client *twitter.Client, text string, inReplyToStatusID int64) {
243	tweet, httpResp, err := client.Statuses.Update(text, &amp;twitter.StatusUpdateParams{
244		InReplyToStatusID: inReplyToStatusID,
245	})
246	if err != nil {
247		fmt.Println(err)
248	}
249	if httpResp.Status != &quot;200 OK&quot; {
250		c.Red(&quot;error: &quot; + httpResp.Status)
251		c.Purple(&quot;maybe twitter has blocked the account, CTRL+C, wait 15 minutes and try again&quot;)
252	}
253	fmt.Print(&quot;tweet posted: &quot;)
254	c.Green(tweet.Text)
255}
256</code></pre>
257
258<h2>3. Flock-Botnet, or how to avoid the Twitter API limitations</h2>
259
260<p>If you ever played with the Twitter API, you will have seen that there are some restrictions and limitations. That means that if your bot have too much posting activity, the account will get blocked for some minutes.</p>
261
262<p>To avoid this limitation, we will deploy a botnet, where each bot will be replying tweets based on the Markov chains probabilities. In this way, when a bot post a tweet reply, the bot falls asleep for 1 minute. In the meantime, the other bots will be processing and replying the other tweets.</p>
263
264<p><img src="img/posts/flock-botnet/flock-botnet-scheme.png" alt="flock-botnet" title="01" /></p>
265
266<h2>3. Putting it all together</h2>
267
268<p>In this demo, we will use only 3 bots (twitter accounts).</p>
269
270<p>The botnet configuration will be in the config.json file:</p>
271
272<pre><code class="language-json">[{
273        &quot;title&quot;: &quot;bot1&quot;,
274        &quot;consumer_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
275        &quot;consumer_secret&quot;: &quot;xxxxxxxxxxxxx&quot;,
276        &quot;access_token_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
277        &quot;access_token_secret&quot;: &quot;xxxxxxxxxxxxx&quot;
278    },
279    {
280        &quot;title&quot;: &quot;bot2&quot;,
281        &quot;consumer_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
282        &quot;consumer_secret&quot;: &quot;xxxxxxxxxxxxx&quot;,
283        &quot;access_token_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
284        &quot;access_token_secret&quot;: &quot;xxxxxxxxxxxxx&quot;
285    },
286    {
287        &quot;title&quot;: &quot;bot3&quot;,
288        &quot;consumer_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
289        &quot;consumer_secret&quot;: &quot;xxxxxxxxxxxxx&quot;,
290        &quot;access_token_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
291        &quot;access_token_secret&quot;: &quot;xxxxxxxxxxxxx&quot;
292    }
293]
294</code></pre>
295
296<p>The complete process will be:</p>
297
298<p><img src="img/posts/flock-botnet/steps.png" alt="flock-botnet" title="01" /></p>
299
300<h2>4. Demo</h2>
301
302<p>We have setted up a small demo with 3 bots. As we have said in the beging of this post, we have used the <a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">“The Critique of Pure Reason”, by Immanuel Kant </a> to generate the Markov chains.</p>
303
304<p>When the botnet is up working, the bots start streaming all the twitter new tweets containing the configured keywords.</p>
305
306<p>Each bot takes a tweet, analyzes the containing words, and generates a reply using the Markov chains previously calculated, and posts the tweet as reply.</p>
307
308<p>Example of terminal view during the flock-botnet execution:</p>
309
310<p><img src="img/posts/flock-botnet/terminal00.png" alt="flock-botnet" title="01" /></p>
311
312<p>Here is an example of the execution:</p>
313
314<p><img src="img/posts/flock-botnet/flock-botnet-demo.gif" alt="flock-botnet" title="Markov chain" /></p>
315
316<p>In the following examples, there are some screenshots that the bots (&ldquo;@andreimarkov&rdquo;, &ldquo;@dodecahedron&rdquo;, &ldquo;@projectNSA&rdquo;) have been replying to some people.</p>
317
318<p><img src="img/posts/flock-botnet/01.png" alt="flock-botnet" title="01" /></p>
319
320<hr>
321
322<p><img src="img/posts/flock-botnet/02.jpeg" alt="flock-botnet" title="02" /></p>
323
324<hr>
325
326<p><img src="img/posts/flock-botnet/03.jpeg" alt="flock-botnet" title="03" /></p>
327
328<hr>
329
330<p><img src="img/posts/flock-botnet/04.jpeg" alt="flock-botnet" title="04" /></p>
331
332<h2>Conclusion</h2>
333
334<p>In this article, we have seen how to build a Twitter botnet with the bots replying tweets with text generated based on Markov chains.</p>
335
336<p>As in this article we have used only 1 grade Markov chains, so the generated text is not really like humans text. But for future projects, a good choice would be combining more grades Markov chains with other text mining techniques.</p>
337
338<p>Twitter API have lots of uses, and in this post we have seen one of them. I hope to be able to write some more articles about other projects arround the Twitter API. For example some Twitter <a href="https://devpost.com/software/projectnsa">network nodes analysis</a>, or some <a href="https://arnaucube.com/hashtagsUsersNetworkPage.html">users &amp; hashtags analysis</a>.</p>
339
340<p>The complete code of this project is able in <a href="https://github.com/arnaucube/flock-botnet">https://github.com/arnaucube/flock-botnet</a></p>
341
342<p>Project page: <a href="http://arnaucube.com/flock-botnet/">http://arnaucube.com/flock-botnet/</a></p>
343
344    </div>
345
346    <footer style="text-align:center; margin-top:100px;margin-bottom:50px;">
347      <div class="container">
348        <br>
349        <a href="/blog">Go to main</a>
350        <br><br>
351        <div class="row">
352          <ul class="list-inline">
353            <li><a href="https://twitter.com/arnaucube"
354                   style="color:gray;text-decoration:none;"
355                   target="_blank">twitter.com/arnaucube</a>
356            </li>
357            <li><a href="https://github.com/arnaucube"
358                   style="color:gray;text-decoration:none;"
359                   target="_blank">github.com/arnaucube</a>
360            </li>
361          </ul>
362        </div>
363        <div class="row" style="display:inline-block;">
364          Blog made with <a href="http://github.com/arnaucube/blogo/"
365                            target="_blank" style="color: gray;text-decoration:none;">Blogo</a>
366        </div>
367      </div>
368    </footer>
369
370    <script>
371    </script>
372    <script src="js/external-links.js"></script>
373    <script>hljs.initHighlightingOnLoad();</script>
374    <script defer src="js/katex/katex.min.js"></script>
375    <script defer src="js/katex/auto-render.min.js"></script>
376    <script>
377      document.addEventListener("DOMContentLoaded", function() {
378        renderMathInElement(document.body, {
379          displayMode: false,
380          // customised options
381          // • auto-render specific keys, e.g.:
382          delimiters: [
383            {left: '$$', right: '$$', display: true},
384            {left: '$', right: '$', display: false},
385            {left: "\\[", right: "\\]", display: true},
386            {left: "\\(", right: "\\)", display: false},
387            {left: "\\begin{equation}", right: "\\end{equation}", display: true}
388          ],
389          // • rendering keys, e.g.:
390          throwOnError : true
391        });
392      });
393
394      ///
395      let theme = localStorage.getItem("theme");
396      if ((theme === "light-theme")||(theme==null)) {
397          theme = "light-theme";
398          document.getElementById("themeSwitcher").checked = false;
399      } else if (theme === "dark-theme") {
400          theme = "dark-theme";
401          document.getElementById("themeSwitcher").checked = true;
402      }
403      document.body.className = theme;
404      localStorage.setItem("theme", theme);
405
406      function switchTheme() {
407        theme = localStorage.getItem("theme");
408        if (theme === "light-theme") {
409          theme = "dark-theme";
410          document.getElementById("themeSwitcher").checked = true;
411        } else {
412          theme = "light-theme";
413          document.getElementById("themeSwitcher").checked = false;
414        }
415        document.body.className = theme;
416        localStorage.setItem("theme", theme);
417
418        console.log(theme);
419      }
420    </script>
421    <script>
422    function tagLinks(tagName) {
423      var tags = document.getElementsByTagName(tagName);
424      for (var i=0, hElem; hElem =  tags[i]; i++) {
425        if (hElem.parentNode.className=="row postThumb") {
426          continue;
427        }
428        hElem.id = hElem.innerHTML.toLowerCase().replace(" ", "-");
429        hElem.innerHTML = "<a style='color:black!important;' href='#"+hElem.id+"'>"+hElem.innerHTML+"</a>";
430      }
431    }
432    tagLinks("h2");
433    tagLinks("h3");
434    tagLinks("h4");
435    tagLinks("h5");
436    </script>
437    <script>
438function generateTOC(tocContainerId = "toc") {
439    const tocContainer = document.getElementById(tocContainerId);
440    if (!tocContainer) return;
441
442    const headings = document.querySelectorAll("h2, h3, h4, h5, h6");
443    if (!headings.length) return;
444
445    // Generate IDs if missing
446    headings.forEach((heading, index) => {
447        if (!heading.id) {
448            heading.id =
449                heading.textContent
450                    .trim()
451                    .toLowerCase()
452                    .replace(/[^\w\s-]/g, "")
453                    .replace(/\s+/g, "-") +
454                "-" +
455                index;
456        }
457    });
458
459    const toc = document.createElement("ul");
460    let currentLevel = 1;
461    let currentList = toc;
462    const stack = [toc];
463
464    headings.forEach((heading) => {
465        const level = parseInt(heading.tagName.substring(1), 10);
466
467        const li = document.createElement("li");
468        const a = document.createElement("a");
469
470        a.href = "#" + heading.id;
471        a.textContent = heading.textContent;
472
473        li.appendChild(a);
474
475        if (level > currentLevel) {
476            for (let i = currentLevel; i < level; i++) {
477                const nestedList = document.createElement("ul");
478
479                const lastLi =
480                    currentList.lastElementChild ||
481                    currentList.appendChild(document.createElement("ul"));
482
483                lastLi.appendChild(nestedList);
484
485                stack.push(nestedList);
486                currentList = nestedList;
487            }
488        } else if (level < currentLevel) {
489            for (let i = currentLevel; i > level; i--) {
490                stack.pop();
491            }
492            currentList = stack[stack.length - 1];
493        }
494
495        currentList.appendChild(li);
496        currentLevel = level;
497    });
498
499    tocContainer.innerHTML = "";
500    tocContainer.appendChild(toc);
501}
502
503document.addEventListener("DOMContentLoaded", () => {
504    generateTOC();
505});
506    </script>
507    <script src="js/mermaid.min.js"></script>
508
509  </body>
510</html>
511