arnaucube
arnaucube/arnaucube.comPublic Self-hosted repository
arnaucube.com/blog/flock-botnet.html
blog/flock-botnet.htmlRaw
  1<!DOCTYPE html>
  2<html lang="en">
  3
  4  <head>
  5    <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
  6    <meta charset="utf-8">
  7    <title> Auto generated tweets from Markov chains - arnaucube - blog</title>
  8    <meta name="title" content=" Auto generated tweets from Markov chains - arnaucube - blog">
  9    <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
 10
 11    <meta property="og:title" content=" Auto generated tweets from Markov chains - arnaucube - blog" />
 12    <meta property="og:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
 13    <meta property="og:url" content="https://arnaucube.com/blog/flock-botnet.html" />
 14    <meta property="og:type" content="article" />
 15    <meta property="og:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png" />
 16    <meta name="twitter:title" content=" Auto generated tweets from Markov chains - arnaucube - blog">
 17    <meta name="twitter:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
 18    <meta name="twitter:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png">
 19    <meta name="twitter:card" content="summary_large_image">
 20    <meta name="author" content="arnaucube">
 21
 22    <link rel="icon" type="image/png" href="img/logoArnauCubeFavicon.png">
 23
 24    <meta name="viewport" content="width=device-width, initial-scale=1">
 25
 26    <link href="css/bootstrap.min.css" rel="stylesheet">
 27    <link rel="stylesheet" href="css/style.css">
 28
 29    <!-- highlightjs -->
 30    <!-- <link rel="stylesheet" href="js/highlightjs/atom-one-dark.css"> -->
 31    <link rel="stylesheet" href="js/highlightjs/atom-one-light.css">
 32    <!-- <link rel="stylesheet" href="js/highlightjs/gruvbox-dark.css"> -->
 33    <script src="js/highlightjs/highlight.pack.js"></script>
 34
 35    <!-- katex -->
 36    <link rel="stylesheet" href="js/katex/katex.min.css">
 37  <link rel="alternate" type="application/rss+xml" title="arnaucube - blog RSS" href="https://arnaucube.com/blog/rss.xml">
 38</head>
 39
 40  <body>
 41
 42    <!-- o_gradient_background" -->
 43    <nav id="mainNav" class="navbar navbar-default navbar-fixed-top"
 44                      style="height:50px;font-size:100%;">
 45      <div class="container">
 46        <div style="float:left;">
 47        <a href="/blog" style="color:#000;display:inline-block;">blog</a>
 48        <span style="margin-right:20px; margin-left:20px;">|</span>
 49        <a href="/blog/notes.html" style="font-size:90%;color:#000;display:inline-block;">notes</a>
 50        </div>
 51        <div style="float:right;">
 52          <a href="/" style="color:#000;display:inline-block;margin-right:10px;">arnaucube.com</a>
 53          <a href="/blog/rss.xml" target="_blank"
 54            type="application/rss+xml" title="subscribe via RSS" aria-label="subscribe via RSS">
 55              <img width="12px;" src="img/rss.png" />
 56          </a>
 57          <div class="onoffswitch" style="margin-left:0px;display:inline-block;" title="change theme">
 58            <input onclick="switchTheme()" type="checkbox" name="onoffswitch" class="onoffswitch-checkbox"
 59              id="themeSwitcher">
 60            <label class="onoffswitch-label" for="themeSwitcher"></label>
 61          </div>
 62        </div>
 63      </div>
 64      <!-- <img style="height:5px; width:100%; margin-top:8px;" src="img/gradient-line.jpg" /> -->
 65    </nav>
 66
 67
 68
 69    <div class="container" style="margin-top:0px;max-width:800px;">
 70      <h1>Auto generated tweets from Markov chains</h1>
 71
 72<p><em>2017-12-29</em></p>
 73
 74<p>The main goal of this article is to explain how to develop a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains. As this is a project to learn, we have made everything from scratch.</p>
 75
 76<p>The idea of mixing twitter bots with Markov chains was in a Twitter conversation with <a href="https://twitter.com/x0rz">@x0rz</a></p>
 77
 78<h2>1. Markov chains</h2>
 79
 80<p>A Markov chain is a sequence of stochastic events (based on probabilities) where the current state of a variable or system is independent of all past states, except the current state.</p>
 81
 82<p><a href="https://en.wikipedia.org/wiki/Markov_chain">https://en.wikipedia.org/wiki/Markov_chain</a></p>
 83
 84<p>In our case, we will use Markov chains to analyze the probabilities that after a word comes another concrete word. So, we will generate some diagram like the following one, but with thousands of words.</p>
 85
 86<p><img src="img/posts/flock-botnet/markovchain.png" alt="flock-botnet" title="Markov chain" /></p>
 87
 88<p>In our case, we need as input some text document with thousands of words, to get a better input data. In this example we have made it with the book &ldquo;The Critique of Pure Reason&rdquo;, by Immanuel Kant (<a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">http://www.gutenberg.org/cache/epub/4280/pg4280.txt</a>), just because is the first book that we have found in .txt format.</p>
 89
 90<h3>1.1 Calculating the Markov chains</h3>
 91
 92<p>First we need to read the text file:</p>
 93
 94<pre><code class="language-go">func readTxt(path string) (string, error) {
 95	data, err := ioutil.ReadFile(path)
 96	if err != nil {
 97		//Do something
 98	}
 99	dataClean := strings.Replace(string(data), &quot;\n&quot;, &quot; &quot;, -1)
100	content := string(dataClean)
101	return content, err
102}
103</code></pre>
104
105<p>To calculate the probabilities of the Markov states, we have made the following function that analyzes the full input text, and stores the Markov states:</p>
106
107<pre><code class="language-go">func calcMarkovStates(words []string) []State {
108	var states []State
109	//count words
110	for i := 0; i &lt; len(words)-1; i++ {
111		var iState int
112		states, iState = addWordToStates(states, words[i])
113		if iState &lt; len(words) {
114			states[iState].NextStates, _ = addWordToStates(states[iState].NextStates, words[i+1])
115		}
116
117		printLoading(i, len(words))
118	}
119
120	//count prob
121	for i := 0; i &lt; len(states); i++ {
122		states[i].Prob = (float64(states[i].Count) / float64(len(words)) * 100)
123		for j := 0; j &lt; len(states[i].NextStates); j++ {
124			states[i].NextStates[j].Prob = (float64(states[i].NextStates[j].Count) / float64(len(words)) * 100)
125		}
126	}
127	fmt.Println(&quot;\ntotal words computed: &quot; + strconv.Itoa(len(words)))
128	return states
129}
130</code></pre>
131
132<p>The <em>printLoading</em> function, is just a simple function to print in the terminal the % of the process done:</p>
133
134<pre><code class="language-go">func printLoading(n int, total int) {
135	var bar []string
136	tantPerFourty := int((float64(n) / float64(total)) * 40)
137	tantPerCent := int((float64(n) / float64(total)) * 100)
138	for i := 0; i &lt; tantPerFourty; i++ {
139		bar = append(bar, &quot;█&quot;)
140	}
141	progressBar := strings.Join(bar, &quot;&quot;)
142	fmt.Printf(&quot;\r &quot; + progressBar + &quot; - &quot; + strconv.Itoa(tantPerCent) + &quot;&quot;)
143}
144</code></pre>
145
146<p><img src="img/posts/flock-botnet/progressbarMarkov.gif" alt="flock-botnet" title="Markov chain" /></p>
147
148<h3>1.2 Generating text from the Markov chains</h3>
149
150<p>To generate the text, we will need a initializer word and the length of the output text to generate. Then, we perform a loop and get words based on the Markov chains probabilities calculated in the previous step.</p>
151
152<pre><code class="language-go">func (markov Markov) generateText(states []State, initWord string, count int) string {
153	var generatedText []string
154	word := initWord
155	generatedText = append(generatedText, word)
156	for i := 0; i &lt; count; i++ {
157		word = getNextMarkovState(states, word)
158		if word == &quot;word no exist on the memory&quot; {
159			return &quot;word no exist on the memory&quot;
160		}
161		generatedText = append(generatedText, word)
162	}
163	text := strings.Join(generatedText, &quot; &quot;)
164	return text
165}
166</code></pre>
167
168<p>To generate the text we need a function that given the markov chains and a word, returns a random probability based word to be the next word following the given word:</p>
169
170<pre><code class="language-go">func getNextMarkovState(states []State, word string) string {
171	iState := -1
172	for i := 0; i &lt; len(states); i++ {
173		if states[i].Word == word {
174			iState = i
175		}
176	}
177	if iState &lt; 0 {
178		return &quot;word no exist on the memory&quot;
179	}
180	var next State
181	next = states[iState].NextStates[0]
182	next.Prob = rand.Float64() * states[iState].Prob
183	for i := 0; i &lt; len(states[iState].NextStates); i++ {
184		if (rand.Float64()*states[iState].NextStates[i].Prob) &gt; next.Prob &amp;&amp; states[iState-1].Word != states[iState].NextStates[i].Word {
185			next = states[iState].NextStates[i]
186		}
187	}
188	return next.Word
189}
190</code></pre>
191
192<h2>2. Twitter API</h2>
193
194<p>To interact with the Twitter API, we will use <strong>go-twitter</strong> library <a href="https://github.com/dghubble/go-twitter">https://github.com/dghubble/go-twitter</a> .</p>
195
196<p>We setup a streaming connection with the Twitter API, we will filter tweets by some words related to our input dataset:</p>
197
198<pre><code class="language-go">func startStreaming(states []State, flock Flock, flockUser *twitter.Client, botScreenName string, keywords []string) {
199	// Convenience Demux demultiplexed stream messages
200	demux := twitter.NewSwitchDemux()
201	demux.Tweet = func(tweet *twitter.Tweet) {
202		if isRT(tweet) == false &amp;&amp; isFromBot(flock, tweet) == false {
203			processTweet(states, flockUser, botScreenName, keywords, tweet)
204		}
205	}
206	demux.DM = func(dm *twitter.DirectMessage) {
207		fmt.Println(dm.SenderID)
208	}
209	demux.Event = func(event *twitter.Event) {
210		fmt.Printf(&quot;%#v\n&quot;, event)
211	}
212
213	fmt.Println(&quot;Starting Stream...&quot;)
214	// FILTER
215	filterParams := &amp;twitter.StreamFilterParams{
216		Track:         keywords,
217		StallWarnings: twitter.Bool(true),
218	}
219	stream, err := flockUser.Streams.Filter(filterParams)
220	if err != nil {
221		log.Fatal(err)
222	}
223	// Receive messages until stopped or stream quits
224	demux.HandleChan(stream.Messages)
225}
226</code></pre>
227
228<p>Then, each time that a new tweet with some of our tracking words is tweeted, we process that tweet and then we generate a reply based on the Markov chains, and we post that reply:</p>
229
230<pre><code class="language-go">func processTweet(states []State, flockUser *twitter.Client, botScreenName string, keywords []string, tweet *twitter.Tweet) {
231	c.Yellow(&quot;bot @&quot; + botScreenName + &quot; - New tweet detected:&quot;)
232	fmt.Println(tweet.Text)
233
234	tweetWords := strings.Split(tweet.Text, &quot; &quot;)
235	generatedText := &quot;word no exist on the memory&quot;
236	for i := 0; i &lt; len(tweetWords) &amp;&amp; generatedText == &quot;word no exist on the memory&quot;; i++ {
237		fmt.Println(strconv.Itoa(i) + &quot; - &quot; + tweetWords[i])
238		generatedText = generateMarkovResponse(states, tweetWords[i])
239	}
240	c.Yellow(&quot;bot @&quot; + botScreenName + &quot; posting response&quot;)
241	fmt.Println(tweet.ID)
242	replyTweet(flockUser, &quot;@&quot;+tweet.User.ScreenName+&quot; &quot;+generatedText, tweet.ID)
243	waitTime(1)
244}
245</code></pre>
246
247<pre><code class="language-go">func replyTweet(client *twitter.Client, text string, inReplyToStatusID int64) {
248	tweet, httpResp, err := client.Statuses.Update(text, &amp;twitter.StatusUpdateParams{
249		InReplyToStatusID: inReplyToStatusID,
250	})
251	if err != nil {
252		fmt.Println(err)
253	}
254	if httpResp.Status != &quot;200 OK&quot; {
255		c.Red(&quot;error: &quot; + httpResp.Status)
256		c.Purple(&quot;maybe twitter has blocked the account, CTRL+C, wait 15 minutes and try again&quot;)
257	}
258	fmt.Print(&quot;tweet posted: &quot;)
259	c.Green(tweet.Text)
260}
261</code></pre>
262
263<h2>3. Flock-Botnet, or how to avoid the Twitter API limitations</h2>
264
265<p>If you ever played with the Twitter API, you will have seen that there are some restrictions and limitations. That means that if your bot have too much posting activity, the account will get blocked for some minutes.</p>
266
267<p>To avoid this limitation, we will deploy a botnet, where each bot will be replying tweets based on the Markov chains probabilities. In this way, when a bot post a tweet reply, the bot falls asleep for 1 minute. In the meantime, the other bots will be processing and replying the other tweets.</p>
268
269<p><img src="img/posts/flock-botnet/flock-botnet-scheme.png" alt="flock-botnet" title="01" /></p>
270
271<h2>3. Putting it all together</h2>
272
273<p>In this demo, we will use only 3 bots (twitter accounts).</p>
274
275<p>The botnet configuration will be in the config.json file:</p>
276
277<pre><code class="language-json">[{
278        &quot;title&quot;: &quot;bot1&quot;,
279        &quot;consumer_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
280        &quot;consumer_secret&quot;: &quot;xxxxxxxxxxxxx&quot;,
281        &quot;access_token_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
282        &quot;access_token_secret&quot;: &quot;xxxxxxxxxxxxx&quot;
283    },
284    {
285        &quot;title&quot;: &quot;bot2&quot;,
286        &quot;consumer_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
287        &quot;consumer_secret&quot;: &quot;xxxxxxxxxxxxx&quot;,
288        &quot;access_token_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
289        &quot;access_token_secret&quot;: &quot;xxxxxxxxxxxxx&quot;
290    },
291    {
292        &quot;title&quot;: &quot;bot3&quot;,
293        &quot;consumer_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
294        &quot;consumer_secret&quot;: &quot;xxxxxxxxxxxxx&quot;,
295        &quot;access_token_key&quot;: &quot;xxxxxxxxxxxxx&quot;,
296        &quot;access_token_secret&quot;: &quot;xxxxxxxxxxxxx&quot;
297    }
298]
299</code></pre>
300
301<p>The complete process will be:</p>
302
303<p><img src="img/posts/flock-botnet/steps.png" alt="flock-botnet" title="01" /></p>
304
305<h2>4. Demo</h2>
306
307<p>We have setted up a small demo with 3 bots. As we have said in the beging of this post, we have used the <a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">“The Critique of Pure Reason”, by Immanuel Kant </a> to generate the Markov chains.</p>
308
309<p>When the botnet is up working, the bots start streaming all the twitter new tweets containing the configured keywords.</p>
310
311<p>Each bot takes a tweet, analyzes the containing words, and generates a reply using the Markov chains previously calculated, and posts the tweet as reply.</p>
312
313<p>Example of terminal view during the flock-botnet execution:</p>
314
315<p><img src="img/posts/flock-botnet/terminal00.png" alt="flock-botnet" title="01" /></p>
316
317<p>Here is an example of the execution:</p>
318
319<p><img src="img/posts/flock-botnet/flock-botnet-demo.gif" alt="flock-botnet" title="Markov chain" /></p>
320
321<p>In the following examples, there are some screenshots that the bots (&ldquo;@andreimarkov&rdquo;, &ldquo;@dodecahedron&rdquo;, &ldquo;@projectNSA&rdquo;) have been replying to some people.</p>
322
323<p><img src="img/posts/flock-botnet/01.png" alt="flock-botnet" title="01" /></p>
324
325<hr>
326
327<p><img src="img/posts/flock-botnet/02.jpeg" alt="flock-botnet" title="02" /></p>
328
329<hr>
330
331<p><img src="img/posts/flock-botnet/03.jpeg" alt="flock-botnet" title="03" /></p>
332
333<hr>
334
335<p><img src="img/posts/flock-botnet/04.jpeg" alt="flock-botnet" title="04" /></p>
336
337<h2>Conclusion</h2>
338
339<p>In this article, we have seen how to build a Twitter botnet with the bots replying tweets with text generated based on Markov chains.</p>
340
341<p>As in this article we have used only 1 grade Markov chains, so the generated text is not really like humans text. But for future projects, a good choice would be combining more grades Markov chains with other text mining techniques.</p>
342
343<p>Twitter API have lots of uses, and in this post we have seen one of them. I hope to be able to write some more articles about other projects arround the Twitter API. For example some Twitter <a href="https://devpost.com/software/projectnsa">network nodes analysis</a>, or some <a href="https://arnaucube.com/hashtagsUsersNetworkPage.html">users &amp; hashtags analysis</a>.</p>
344
345<p>The complete code of this project is able in <a href="https://github.com/arnaucube/flock-botnet">https://github.com/arnaucube/flock-botnet</a></p>
346
347<p>Project page: <a href="http://arnaucube.com/flock-botnet/">http://arnaucube.com/flock-botnet/</a></p>
348
349    </div>
350
351    <footer style="text-align:center; margin-top:100px;margin-bottom:50px;">
352      <div class="container">
353        <br>
354        <a href="/blog">Go to main</a>
355        <br><br>
356        <div class="row">
357          <ul class="list-inline">
358            <li><a href="https://twitter.com/arnaucube"
359                   style="color:gray;text-decoration:none;"
360                   target="_blank">twitter.com/arnaucube</a>
361            </li>
362            <li><a href="https://github.com/arnaucube"
363                   style="color:gray;text-decoration:none;"
364                   target="_blank">github.com/arnaucube</a>
365            </li>
366          </ul>
367        </div>
368        <div class="row" style="display:inline-block;">
369          Blog made with <a href="http://github.com/arnaucube/blogo/"
370                            target="_blank" style="color: gray;text-decoration:none;">Blogo</a>
371        </div>
372      </div>
373    </footer>
374
375    <script>
376    </script>
377    <script src="js/external-links.js"></script>
378    <script>hljs.initHighlightingOnLoad();</script>
379    <script defer src="js/katex/katex.min.js"></script>
380    <script defer src="js/katex/auto-render.min.js"></script>
381    <script>
382      document.addEventListener("DOMContentLoaded", function() {
383        renderMathInElement(document.body, {
384          displayMode: false,
385          // customised options
386          // • auto-render specific keys, e.g.:
387          delimiters: [
388            {left: '$$', right: '$$', display: true},
389            {left: '$', right: '$', display: false},
390            {left: "\\[", right: "\\]", display: true},
391            {left: "\\(", right: "\\)", display: false},
392            {left: "\\begin{equation}", right: "\\end{equation}", display: true}
393          ],
394          // • rendering keys, e.g.:
395          throwOnError : true
396        });
397      });
398
399      ///
400      let theme = localStorage.getItem("theme");
401      if ((theme === "light-theme")||(theme==null)) {
402          theme = "light-theme";
403          document.getElementById("themeSwitcher").checked = false;
404      } else if (theme === "dark-theme") {
405          theme = "dark-theme";
406          document.getElementById("themeSwitcher").checked = true;
407      }
408      document.body.className = theme;
409      localStorage.setItem("theme", theme);
410
411      function switchTheme() {
412        theme = localStorage.getItem("theme");
413        if (theme === "light-theme") {
414          theme = "dark-theme";
415          document.getElementById("themeSwitcher").checked = true;
416        } else {
417          theme = "light-theme";
418          document.getElementById("themeSwitcher").checked = false;
419        }
420        document.body.className = theme;
421        localStorage.setItem("theme", theme);
422
423        console.log(theme);
424      }
425    </script>
426    <script>
427    function tagLinks(tagName) {
428      var tags = document.getElementsByTagName(tagName);
429      for (var i=0, hElem; hElem =  tags[i]; i++) {
430        if (hElem.parentNode.className=="row postThumb") {
431          continue;
432        }
433        hElem.id = hElem.innerHTML.toLowerCase().replace(" ", "-");
434        hElem.innerHTML = "<a style='color:black!important;' href='#"+hElem.id+"'>"+hElem.innerHTML+"</a>";
435      }
436    }
437    tagLinks("h2");
438    tagLinks("h3");
439    tagLinks("h4");
440    tagLinks("h5");
441    </script>
442    <script>
443function generateTOC(tocContainerId = "toc") {
444    const tocContainer = document.getElementById(tocContainerId);
445    if (!tocContainer) return;
446
447    const headings = document.querySelectorAll("h2, h3, h4, h5, h6");
448    if (!headings.length) return;
449
450    // Generate IDs if missing
451    headings.forEach((heading, index) => {
452        if (!heading.id) {
453            heading.id =
454                heading.textContent
455                    .trim()
456                    .toLowerCase()
457                    .replace(/[^\w\s-]/g, "")
458                    .replace(/\s+/g, "-") +
459                "-" +
460                index;
461        }
462    });
463
464    const toc = document.createElement("ul");
465    let currentLevel = 1;
466    let currentList = toc;
467    const stack = [toc];
468
469    headings.forEach((heading) => {
470        const level = parseInt(heading.tagName.substring(1), 10);
471
472        const li = document.createElement("li");
473        const a = document.createElement("a");
474
475        a.href = "#" + heading.id;
476        a.textContent = heading.textContent;
477
478        li.appendChild(a);
479
480        if (level > currentLevel) {
481            for (let i = currentLevel; i < level; i++) {
482                const nestedList = document.createElement("ul");
483
484                const lastLi =
485                    currentList.lastElementChild ||
486                    currentList.appendChild(document.createElement("ul"));
487
488                lastLi.appendChild(nestedList);
489
490                stack.push(nestedList);
491                currentList = nestedList;
492            }
493        } else if (level < currentLevel) {
494            for (let i = currentLevel; i > level; i--) {
495                stack.pop();
496            }
497            currentList = stack[stack.length - 1];
498        }
499
500        currentList.appendChild(li);
501        currentLevel = level;
502    });
503
504    tocContainer.innerHTML = "";
505    tocContainer.appendChild(toc);
506}
507
508document.addEventListener("DOMContentLoaded", () => {
509    generateTOC();
510});
511    </script>
512    <script src="js/mermaid.min.js"></script>
513
514  </body>
515</html>
516