arnaucube.com/blogo-input/posts/flock-botnet.md
blogo-input/posts/flock-botnet.mdRaw
1# Auto generated tweets from Markov chains
2*2017-12-29*
3
4The main goal of this article is to explain how to develop a twitter botnet with autonomous bots replying tweets with text generated based on probabilities in Markov chains. As this is a project to learn, we have made everything from scratch.
5
6The idea of mixing twitter bots with Markov chains was in a Twitter conversation with [@x0rz](https://twitter.com/x0rz)
7
8## 1. Markov chains
9A Markov chain is a sequence of stochastic events (based on probabilities) where the current state of a variable or system is independent of all past states, except the current state.
10
11https://en.wikipedia.org/wiki/Markov_chain
12
13In our case, we will use Markov chains to analyze the probabilities that after a word comes another concrete word. So, we will generate some diagram like the following one, but with thousands of words.
14
15
16
17In our case, we need as input some text document with thousands of words, to get a better input data. In this example we have made it with the book "The Critique of Pure Reason", by Immanuel Kant (http://www.gutenberg.org/cache/epub/4280/pg4280.txt), just because is the first book that we have found in .txt format.
18
19### 1.1 Calculating the Markov chains
20First we need to read the text file:
21
22```go
23func readTxt(path string) (string, error) {
24 data, err := ioutil.ReadFile(path)
25 if err != nil {
26 //Do something
27 }
28 dataClean := strings.Replace(string(data), "\n", " ", -1)
29 content := string(dataClean)
30 return content, err
31}
32```
33
34To calculate the probabilities of the Markov states, we have made the following function that analyzes the full input text, and stores the Markov states:
35
36```go
37func calcMarkovStates(words []string) []State {
38 var states []State
39 //count words
40 for i := 0; i < len(words)-1; i++ {
41 var iState int
42 states, iState = addWordToStates(states, words[i])
43 if iState < len(words) {
44 states[iState].NextStates, _ = addWordToStates(states[iState].NextStates, words[i+1])
45 }
46
47 printLoading(i, len(words))
48 }
49
50 //count prob
51 for i := 0; i < len(states); i++ {
52 states[i].Prob = (float64(states[i].Count) / float64(len(words)) * 100)
53 for j := 0; j < len(states[i].NextStates); j++ {
54 states[i].NextStates[j].Prob = (float64(states[i].NextStates[j].Count) / float64(len(words)) * 100)
55 }
56 }
57 fmt.Println("\ntotal words computed: " + strconv.Itoa(len(words)))
58 return states
59}
60```
61
62The *printLoading* function, is just a simple function to print in the terminal the % of the process done:
63
64```go
65func printLoading(n int, total int) {
66 var bar []string
67 tantPerFourty := int((float64(n) / float64(total)) * 40)
68 tantPerCent := int((float64(n) / float64(total)) * 100)
69 for i := 0; i < tantPerFourty; i++ {
70 bar = append(bar, "█")
71 }
72 progressBar := strings.Join(bar, "")
73 fmt.Printf("\r " + progressBar + " - " + strconv.Itoa(tantPerCent) + "")
74}
75```
76
77
78
79
80### 1.2 Generating text from the Markov chains
81To generate the text, we will need a initializer word and the length of the output text to generate. Then, we perform a loop and get words based on the Markov chains probabilities calculated in the previous step.
82
83```go
84func (markov Markov) generateText(states []State, initWord string, count int) string {
85 var generatedText []string
86 word := initWord
87 generatedText = append(generatedText, word)
88 for i := 0; i < count; i++ {
89 word = getNextMarkovState(states, word)
90 if word == "word no exist on the memory" {
91 return "word no exist on the memory"
92 }
93 generatedText = append(generatedText, word)
94 }
95 text := strings.Join(generatedText, " ")
96 return text
97}
98```
99
100To generate the text we need a function that given the markov chains and a word, returns a random probability based word to be the next word following the given word:
101
102```go
103func getNextMarkovState(states []State, word string) string {
104 iState := -1
105 for i := 0; i < len(states); i++ {
106 if states[i].Word == word {
107 iState = i
108 }
109 }
110 if iState < 0 {
111 return "word no exist on the memory"
112 }
113 var next State
114 next = states[iState].NextStates[0]
115 next.Prob = rand.Float64() * states[iState].Prob
116 for i := 0; i < len(states[iState].NextStates); i++ {
117 if (rand.Float64()*states[iState].NextStates[i].Prob) > next.Prob && states[iState-1].Word != states[iState].NextStates[i].Word {
118 next = states[iState].NextStates[i]
119 }
120 }
121 return next.Word
122}
123```
124
125
126## 2. Twitter API
127To interact with the Twitter API, we will use **go-twitter** library https://github.com/dghubble/go-twitter .
128
129We setup a streaming connection with the Twitter API, we will filter tweets by some words related to our input dataset:
130
131```go
132func startStreaming(states []State, flock Flock, flockUser *twitter.Client, botScreenName string, keywords []string) {
133 // Convenience Demux demultiplexed stream messages
134 demux := twitter.NewSwitchDemux()
135 demux.Tweet = func(tweet *twitter.Tweet) {
136 if isRT(tweet) == false && isFromBot(flock, tweet) == false {
137 processTweet(states, flockUser, botScreenName, keywords, tweet)
138 }
139 }
140 demux.DM = func(dm *twitter.DirectMessage) {
141 fmt.Println(dm.SenderID)
142 }
143 demux.Event = func(event *twitter.Event) {
144 fmt.Printf("%#v\n", event)
145 }
146
147 fmt.Println("Starting Stream...")
148 // FILTER
149 filterParams := &twitter.StreamFilterParams{
150 Track: keywords,
151 StallWarnings: twitter.Bool(true),
152 }
153 stream, err := flockUser.Streams.Filter(filterParams)
154 if err != nil {
155 log.Fatal(err)
156 }
157 // Receive messages until stopped or stream quits
158 demux.HandleChan(stream.Messages)
159}
160```
161
162Then, each time that a new tweet with some of our tracking words is tweeted, we process that tweet and then we generate a reply based on the Markov chains, and we post that reply:
163```go
164func processTweet(states []State, flockUser *twitter.Client, botScreenName string, keywords []string, tweet *twitter.Tweet) {
165 c.Yellow("bot @" + botScreenName + " - New tweet detected:")
166 fmt.Println(tweet.Text)
167
168 tweetWords := strings.Split(tweet.Text, " ")
169 generatedText := "word no exist on the memory"
170 for i := 0; i < len(tweetWords) && generatedText == "word no exist on the memory"; i++ {
171 fmt.Println(strconv.Itoa(i) + " - " + tweetWords[i])
172 generatedText = generateMarkovResponse(states, tweetWords[i])
173 }
174 c.Yellow("bot @" + botScreenName + " posting response")
175 fmt.Println(tweet.ID)
176 replyTweet(flockUser, "@"+tweet.User.ScreenName+" "+generatedText, tweet.ID)
177 waitTime(1)
178}
179```
180
181```go
182func replyTweet(client *twitter.Client, text string, inReplyToStatusID int64) {
183 tweet, httpResp, err := client.Statuses.Update(text, &twitter.StatusUpdateParams{
184 InReplyToStatusID: inReplyToStatusID,
185 })
186 if err != nil {
187 fmt.Println(err)
188 }
189 if httpResp.Status != "200 OK" {
190 c.Red("error: " + httpResp.Status)
191 c.Purple("maybe twitter has blocked the account, CTRL+C, wait 15 minutes and try again")
192 }
193 fmt.Print("tweet posted: ")
194 c.Green(tweet.Text)
195}
196```
197
198## 3. Flock-Botnet, or how to avoid the Twitter API limitations
199If you ever played with the Twitter API, you will have seen that there are some restrictions and limitations. That means that if your bot have too much posting activity, the account will get blocked for some minutes.
200
201To avoid this limitation, we will deploy a botnet, where each bot will be replying tweets based on the Markov chains probabilities. In this way, when a bot post a tweet reply, the bot falls asleep for 1 minute. In the meantime, the other bots will be processing and replying the other tweets.
202
203
204
205
206## 3. Putting it all together
207
208In this demo, we will use only 3 bots (twitter accounts).
209
210The botnet configuration will be in the config.json file:
211
212```json
213[{
214 "title": "bot1",
215 "consumer_key": "xxxxxxxxxxxxx",
216 "consumer_secret": "xxxxxxxxxxxxx",
217 "access_token_key": "xxxxxxxxxxxxx",
218 "access_token_secret": "xxxxxxxxxxxxx"
219 },
220 {
221 "title": "bot2",
222 "consumer_key": "xxxxxxxxxxxxx",
223 "consumer_secret": "xxxxxxxxxxxxx",
224 "access_token_key": "xxxxxxxxxxxxx",
225 "access_token_secret": "xxxxxxxxxxxxx"
226 },
227 {
228 "title": "bot3",
229 "consumer_key": "xxxxxxxxxxxxx",
230 "consumer_secret": "xxxxxxxxxxxxx",
231 "access_token_key": "xxxxxxxxxxxxx",
232 "access_token_secret": "xxxxxxxxxxxxx"
233 }
234]
235```
236
237The complete process will be:
238
239
240
241
242## 4. Demo
243We have setted up a small demo with 3 bots. As we have said in the beging of this post, we have used the [“The Critique of Pure Reason”, by Immanuel Kant ](http://www.gutenberg.org/cache/epub/4280/pg4280.txt) to generate the Markov chains.
244
245When the botnet is up working, the bots start streaming all the twitter new tweets containing the configured keywords.
246
247Each bot takes a tweet, analyzes the containing words, and generates a reply using the Markov chains previously calculated, and posts the tweet as reply.
248
249Example of terminal view during the flock-botnet execution:
250
251
252
253Here is an example of the execution:
254
255
256
257In the following examples, there are some screenshots that the bots ("@andreimarkov", "@dodecahedron", "@projectNSA") have been replying to some people.
258
259
260
261---
262
263
264
265---
266
267
268
269---
270
271
272
273
274
275
276## Conclusion
277In this article, we have seen how to build a Twitter botnet with the bots replying tweets with text generated based on Markov chains.
278
279As in this article we have used only 1 grade Markov chains, so the generated text is not really like humans text. But for future projects, a good choice would be combining more grades Markov chains with other text mining techniques.
280
281Twitter API have lots of uses, and in this post we have seen one of them. I hope to be able to write some more articles about other projects arround the Twitter API. For example some Twitter [network nodes analysis](https://devpost.com/software/projectnsa), or some [users & hashtags analysis](https://arnaucube.com/hashtagsUsersNetworkPage.html).
282
283The complete code of this project is able in https://github.com/arnaucube/flock-botnet
284
285Project page: http://arnaucube.com/flock-botnet/
286