arnaucube.com/public/flock-botnet.html
public/flock-botnet.htmlRaw
1<!DOCTYPE html>
2<html lang="en">
3
4 <head>
5 <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
6 <meta charset="utf-8">
7 <title>Auto generated tweets from Markov chains - arnaucube</title>
8 <meta name="title" content="Auto generated tweets from Markov chains - arnaucube">
9 <meta name="description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
10
11 <meta property="og:title" content="Auto generated tweets from Markov chains - arnaucube" />
12 <meta property="og:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains" />
13 <meta property="og:url" content="https://arnaucube.com/blog/flock-botnet.html" />
14 <meta property="og:type" content="article" />
15 <meta property="og:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png" />
16 <meta name="twitter:title" content="Auto generated tweets from Markov chains - arnaucube">
17 <meta name="twitter:description" content="Developing a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains">
18 <meta name="twitter:image" content="https://arnaucube.com/blog/img/posts/flock-botnet/markovchain.png">
19 <meta name="twitter:card" content="summary_large_image">
20 <meta name="author" content="arnaucube">
21
22 <meta name="viewport" content="width=device-width, initial-scale=1">
23
24 <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous">
25 <link rel="stylesheet" href="css/style.css">
26
27 <!-- highlightjs -->
28 <!-- <link rel="stylesheet" href="js/highlightjs/atom-one-dark.css"> -->
29 <link rel="stylesheet" href="js/highlightjs/gruvbox-dark.css">
30 <script src="js/highlightjs/highlight.pack.js"></script>
31
32 <!-- katex -->
33 <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.13.11/dist/katex.min.css" integrity="sha384-Um5gpz1odJg5Z4HAmzPtgZKdTBHZdw8S29IecapCSB31ligYPhHQZMIlWLYQGVoc" crossorigin="anonymous">
34 </head>
35
36 <body>
37
38 <!-- o_gradient_background" -->
39 <nav id="mainNav" class="navbar navbar-default navbar-fixed-top"
40 style="height:50px;font-size:130%;">
41 <div class="container">
42 <a href="/blog" style="color:#000;">Blog index</a>
43 <a href="/" style="color:#000;float:right;">arnaucube.com</a>
44 </div>
45 <img style="height:5px; width:100%; margin-top:8px;" src="img/gradient-line.jpg" />
46 </nav>
47
48
49
50 <div class="container" style="margin-top:40px;max-width:800px;">
51 <h1>Auto generated tweets from Markov chains</h1>
52
53<p><em>2017-12-29</em></p>
54
55<p>The main goal of this article is to explain how to develop a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains. As this is a project to learn, we have made everything from scratch.</p>
56
57<p>The idea of mixing twitter bots with Markov chains was in a Twitter conversation with <a href="https://twitter.com/x0rz">@x0rz</a></p>
58
59<h2>1. Markov chains</h2>
60
61<p>A Markov chain is a sequence of stochastic events (based on probabilities) where the current state of a variable or system is independent of all past states, except the current state.</p>
62
63<p><a href="https://en.wikipedia.org/wiki/Markov_chain">https://en.wikipedia.org/wiki/Markov_chain</a></p>
64
65<p>In our case, we will use Markov chains to analyze the probabilities that after a word comes another concrete word. So, we will generate some diagram like the following one, but with thousands of words.</p>
66
67<p><img src="img/posts/flock-botnet/markovchain.png" alt="flock-botnet" title="Markov chain" /></p>
68
69<p>In our case, we need as input some text document with thousands of words, to get a better input data. In this example we have made it with the book “The Critique of Pure Reason”, by Immanuel Kant (<a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">http://www.gutenberg.org/cache/epub/4280/pg4280.txt</a>), just because is the first book that we have found in .txt format.</p>
70
71<h3>1.1 Calculating the Markov chains</h3>
72
73<p>First we need to read the text file:</p>
74
75<pre><code class="language-go">func readTxt(path string) (string, error) {
76 data, err := ioutil.ReadFile(path)
77 if err != nil {
78 //Do something
79 }
80 dataClean := strings.Replace(string(data), "\n", " ", -1)
81 content := string(dataClean)
82 return content, err
83}
84</code></pre>
85
86<p>To calculate the probabilities of the Markov states, we have made the following function that analyzes the full input text, and stores the Markov states:</p>
87
88<pre><code class="language-go">func calcMarkovStates(words []string) []State {
89 var states []State
90 //count words
91 for i := 0; i < len(words)-1; i++ {
92 var iState int
93 states, iState = addWordToStates(states, words[i])
94 if iState < len(words) {
95 states[iState].NextStates, _ = addWordToStates(states[iState].NextStates, words[i+1])
96 }
97
98 printLoading(i, len(words))
99 }
100
101 //count prob
102 for i := 0; i < len(states); i++ {
103 states[i].Prob = (float64(states[i].Count) / float64(len(words)) * 100)
104 for j := 0; j < len(states[i].NextStates); j++ {
105 states[i].NextStates[j].Prob = (float64(states[i].NextStates[j].Count) / float64(len(words)) * 100)
106 }
107 }
108 fmt.Println("\ntotal words computed: " + strconv.Itoa(len(words)))
109 return states
110}
111</code></pre>
112
113<p>The <em>printLoading</em> function, is just a simple function to print in the terminal the % of the process done:</p>
114
115<pre><code class="language-go">func printLoading(n int, total int) {
116 var bar []string
117 tantPerFourty := int((float64(n) / float64(total)) * 40)
118 tantPerCent := int((float64(n) / float64(total)) * 100)
119 for i := 0; i < tantPerFourty; i++ {
120 bar = append(bar, "█")
121 }
122 progressBar := strings.Join(bar, "")
123 fmt.Printf("\r " + progressBar + " - " + strconv.Itoa(tantPerCent) + "")
124}
125</code></pre>
126
127<p><img src="img/posts/flock-botnet/progressbarMarkov.gif" alt="flock-botnet" title="Markov chain" /></p>
128
129<h3>1.2 Generating text from the Markov chains</h3>
130
131<p>To generate the text, we will need a initializer word and the length of the output text to generate. Then, we perform a loop and get words based on the Markov chains probabilities calculated in the previous step.</p>
132
133<pre><code class="language-go">func (markov Markov) generateText(states []State, initWord string, count int) string {
134 var generatedText []string
135 word := initWord
136 generatedText = append(generatedText, word)
137 for i := 0; i < count; i++ {
138 word = getNextMarkovState(states, word)
139 if word == "word no exist on the memory" {
140 return "word no exist on the memory"
141 }
142 generatedText = append(generatedText, word)
143 }
144 text := strings.Join(generatedText, " ")
145 return text
146}
147</code></pre>
148
149<p>To generate the text we need a function that given the markov chains and a word, returns a random probability based word to be the next word following the given word:</p>
150
151<pre><code class="language-go">func getNextMarkovState(states []State, word string) string {
152 iState := -1
153 for i := 0; i < len(states); i++ {
154 if states[i].Word == word {
155 iState = i
156 }
157 }
158 if iState < 0 {
159 return "word no exist on the memory"
160 }
161 var next State
162 next = states[iState].NextStates[0]
163 next.Prob = rand.Float64() * states[iState].Prob
164 for i := 0; i < len(states[iState].NextStates); i++ {
165 if (rand.Float64()*states[iState].NextStates[i].Prob) > next.Prob && states[iState-1].Word != states[iState].NextStates[i].Word {
166 next = states[iState].NextStates[i]
167 }
168 }
169 return next.Word
170}
171</code></pre>
172
173<h2>2. Twitter API</h2>
174
175<p>To interact with the Twitter API, we will use <strong>go-twitter</strong> library <a href="https://github.com/dghubble/go-twitter">https://github.com/dghubble/go-twitter</a> .</p>
176
177<p>We setup a streaming connection with the Twitter API, we will filter tweets by some words related to our input dataset:</p>
178
179<pre><code class="language-go">func startStreaming(states []State, flock Flock, flockUser *twitter.Client, botScreenName string, keywords []string) {
180 // Convenience Demux demultiplexed stream messages
181 demux := twitter.NewSwitchDemux()
182 demux.Tweet = func(tweet *twitter.Tweet) {
183 if isRT(tweet) == false && isFromBot(flock, tweet) == false {
184 processTweet(states, flockUser, botScreenName, keywords, tweet)
185 }
186 }
187 demux.DM = func(dm *twitter.DirectMessage) {
188 fmt.Println(dm.SenderID)
189 }
190 demux.Event = func(event *twitter.Event) {
191 fmt.Printf("%#v\n", event)
192 }
193
194 fmt.Println("Starting Stream...")
195 // FILTER
196 filterParams := &twitter.StreamFilterParams{
197 Track: keywords,
198 StallWarnings: twitter.Bool(true),
199 }
200 stream, err := flockUser.Streams.Filter(filterParams)
201 if err != nil {
202 log.Fatal(err)
203 }
204 // Receive messages until stopped or stream quits
205 demux.HandleChan(stream.Messages)
206}
207</code></pre>
208
209<p>Then, each time that a new tweet with some of our tracking words is tweeted, we process that tweet and then we generate a reply based on the Markov chains, and we post that reply:</p>
210
211<pre><code class="language-go">func processTweet(states []State, flockUser *twitter.Client, botScreenName string, keywords []string, tweet *twitter.Tweet) {
212 c.Yellow("bot @" + botScreenName + " - New tweet detected:")
213 fmt.Println(tweet.Text)
214
215 tweetWords := strings.Split(tweet.Text, " ")
216 generatedText := "word no exist on the memory"
217 for i := 0; i < len(tweetWords) && generatedText == "word no exist on the memory"; i++ {
218 fmt.Println(strconv.Itoa(i) + " - " + tweetWords[i])
219 generatedText = generateMarkovResponse(states, tweetWords[i])
220 }
221 c.Yellow("bot @" + botScreenName + " posting response")
222 fmt.Println(tweet.ID)
223 replyTweet(flockUser, "@"+tweet.User.ScreenName+" "+generatedText, tweet.ID)
224 waitTime(1)
225}
226</code></pre>
227
228<pre><code class="language-go">func replyTweet(client *twitter.Client, text string, inReplyToStatusID int64) {
229 tweet, httpResp, err := client.Statuses.Update(text, &twitter.StatusUpdateParams{
230 InReplyToStatusID: inReplyToStatusID,
231 })
232 if err != nil {
233 fmt.Println(err)
234 }
235 if httpResp.Status != "200 OK" {
236 c.Red("error: " + httpResp.Status)
237 c.Purple("maybe twitter has blocked the account, CTRL+C, wait 15 minutes and try again")
238 }
239 fmt.Print("tweet posted: ")
240 c.Green(tweet.Text)
241}
242</code></pre>
243
244<h2>3. Flock-Botnet, or how to avoid the Twitter API limitations</h2>
245
246<p>If you ever played with the Twitter API, you will have seen that there are some restrictions and limitations. That means that if your bot have too much posting activity, the account will get blocked for some minutes.</p>
247
248<p>To avoid this limitation, we will deploy a botnet, where each bot will be replying tweets based on the Markov chains probabilities. In this way, when a bot post a tweet reply, the bot falls asleep for 1 minute. In the meantime, the other bots will be processing and replying the other tweets.</p>
249
250<p><img src="img/posts/flock-botnet/flock-botnet-scheme.png" alt="flock-botnet" title="01" /></p>
251
252<h2>3. Putting it all together</h2>
253
254<p>In this demo, we will use only 3 bots (twitter accounts).</p>
255
256<p>The botnet configuration will be in the config.json file:</p>
257
258<pre><code class="language-json">[{
259 "title": "bot1",
260 "consumer_key": "xxxxxxxxxxxxx",
261 "consumer_secret": "xxxxxxxxxxxxx",
262 "access_token_key": "xxxxxxxxxxxxx",
263 "access_token_secret": "xxxxxxxxxxxxx"
264 },
265 {
266 "title": "bot2",
267 "consumer_key": "xxxxxxxxxxxxx",
268 "consumer_secret": "xxxxxxxxxxxxx",
269 "access_token_key": "xxxxxxxxxxxxx",
270 "access_token_secret": "xxxxxxxxxxxxx"
271 },
272 {
273 "title": "bot3",
274 "consumer_key": "xxxxxxxxxxxxx",
275 "consumer_secret": "xxxxxxxxxxxxx",
276 "access_token_key": "xxxxxxxxxxxxx",
277 "access_token_secret": "xxxxxxxxxxxxx"
278 }
279]
280</code></pre>
281
282<p>The complete process will be:</p>
283
284<p><img src="img/posts/flock-botnet/steps.png" alt="flock-botnet" title="01" /></p>
285
286<h2>4. Demo</h2>
287
288<p>We have setted up a small demo with 3 bots. As we have said in the beging of this post, we have used the <a href="http://www.gutenberg.org/cache/epub/4280/pg4280.txt">“The Critique of Pure Reason”, by Immanuel Kant </a> to generate the Markov chains.</p>
289
290<p>When the botnet is up working, the bots start streaming all the twitter new tweets containing the configured keywords.</p>
291
292<p>Each bot takes a tweet, analyzes the containing words, and generates a reply using the Markov chains previously calculated, and posts the tweet as reply.</p>
293
294<p>Example of terminal view during the flock-botnet execution:</p>
295
296<p><img src="img/posts/flock-botnet/terminal00.png" alt="flock-botnet" title="01" /></p>
297
298<p>Here is an example of the execution:</p>
299
300<p><img src="img/posts/flock-botnet/flock-botnet-demo.gif" alt="flock-botnet" title="Markov chain" /></p>
301
302<p>In the following examples, there are some screenshots that the bots (“@andreimarkov”, “@dodecahedron”, “@projectNSA”) have been replying to some people.</p>
303
304<p><img src="img/posts/flock-botnet/01.png" alt="flock-botnet" title="01" /></p>
305
306<hr>
307
308<p><img src="img/posts/flock-botnet/02.jpeg" alt="flock-botnet" title="02" /></p>
309
310<hr>
311
312<p><img src="img/posts/flock-botnet/03.jpeg" alt="flock-botnet" title="03" /></p>
313
314<hr>
315
316<p><img src="img/posts/flock-botnet/04.jpeg" alt="flock-botnet" title="04" /></p>
317
318<h2>Conclusion</h2>
319
320<p>In this article, we have seen how to build a Twitter botnet with the bots replying tweets with text generated based on Markov chains.</p>
321
322<p>As in this article we have used only 1 grade Markov chains, so the generated text is not really like humans text. But for future projects, a good choice would be combining more grades Markov chains with other text mining techniques.</p>
323
324<p>Twitter API have lots of uses, and in this post we have seen one of them. I hope to be able to write some more articles about other projects arround the Twitter API. For example some Twitter <a href="https://devpost.com/software/projectnsa">network nodes analysis</a>, or some <a href="https://arnaucube.com/hashtagsUsersNetworkPage.html">users & hashtags analysis</a>.</p>
325
326<p>The complete code of this project is able in <a href="https://github.com/arnaucube/flock-botnet">https://github.com/arnaucube/flock-botnet</a></p>
327
328<p>Project page: <a href="http://arnaucube.com/flock-botnet/">http://arnaucube.com/flock-botnet/</a></p>
329
330 </div>
331
332 <footer style="text-align:center; margin-top:100px;margin-bottom:50px;">
333 <div class="container">
334 <div class="row">
335 <ul class="list-inline">
336 <li><a href="https://twitter.com/arnaucube"
337 style="color:gray;text-decoration:none;"
338 target="_blank">twitter.com/arnaucube</a>
339 </li>
340 <li><a href="https://github.com/arnaucube"
341 style="color:gray;text-decoration:none;"
342 target="_blank">github.com/arnaucube</a>
343 </li>
344 </ul>
345 </div>
346 <div class="row" style="display:inline-block;">
347 Blog made with <a href="http://github.com/arnaucube/blogo/"
348 target="_blank" style="color: gray;text-decoration:none;">Blogo</a>
349 </div>
350 </div>
351 </footer>
352
353 <script>
354 </script>
355 <script src="js/external-links.js"></script>
356 <script>hljs.initHighlightingOnLoad();</script>
357 <script defer src="https://cdn.jsdelivr.net/npm/katex@0.13.11/dist/katex.min.js" integrity="sha384-YNHdsYkH6gMx9y3mRkmcJ2mFUjTd0qNQQvY9VYZgQd7DcN7env35GzlmFaZ23JGp" crossorigin="anonymous"></script>
358 <script defer src="https://cdn.jsdelivr.net/npm/katex@0.13.11/dist/contrib/auto-render.min.js" integrity="sha384-vZTG03m+2yp6N6BNi5iM4rW4oIwk5DfcNdFfxkk9ZWpDriOkXX8voJBFrAO7MpVl" crossorigin="anonymous"></script>
359 <script>
360 document.addEventListener("DOMContentLoaded", function() {
361 renderMathInElement(document.body, {
362 displayMode: false,
363 // customised options
364 // • auto-render specific keys, e.g.:
365 delimiters: [
366 {left: '$$', right: '$$', display: true},
367 {left: '$', right: '$', display: false},
368 ],
369 // • rendering keys, e.g.:
370 throwOnError : true
371 });
372 });
373 </script>
374 <script src="https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.min.js"></script>
375
376
377 </body>
378</html>
379