You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

311 lines
10 KiB

Allow serving API only via new cli command - Add new command to the cli/node: `serveapi` that alows serving the API just by connecting to the PostgreSQL database. The mode flag should me passed in order to select whether we are connecting to a synchronizer database or a coordinator database. If `coord` is chosen as mode, the coordinator endpoints can be activated in order to allow inserting l2txs and authorizations into the L2DB. Summary of the implementation details - New SQL table with 3 columns (plus `item_id` pk). The table only contains a single row with `item_id` = 1. Columns: - state: historydb.StateAPI in JSON. This is the struct that is served via the `/state` API endpoint. The node will periodically update this struct and store it int he DB. The api server will query it from the DB to serve it. - config: historydb.NodeConfig in JSON. This struct contains node configuration parameters that the API needs to be aware of. It's updated once every time the node starts. - constants: historydb.Constants in JSON. This struct contains all the hermez network constants gathered via the ethereum client by the node. It's written once every time the node starts. - The HistoryDB contains methods to get and update each one of these columns individually. - The HistoryDB contains all methods that query the DB and prepare objects that will appear in the StateAPI endpoint. - The configuration used in for the `serveapi` cli/node command is defined in `config.APIServer`, and is a subset of `node.Config` in order to allow reusing the same configuration file of the node if desired. - A new object is introduced in the api: `StateAPIUpdater`, which contains all the necessary information to update the StateAPI in the DB periodically by the node. - Moved the types `SCConsts`, `SCVariables` and `SCVariablesPtr` from `syncrhonizer` to `common` for convenience.
3 years ago
Allow serving API only via new cli command - Add new command to the cli/node: `serveapi` that alows serving the API just by connecting to the PostgreSQL database. The mode flag should me passed in order to select whether we are connecting to a synchronizer database or a coordinator database. If `coord` is chosen as mode, the coordinator endpoints can be activated in order to allow inserting l2txs and authorizations into the L2DB. Summary of the implementation details - New SQL table with 3 columns (plus `item_id` pk). The table only contains a single row with `item_id` = 1. Columns: - state: historydb.StateAPI in JSON. This is the struct that is served via the `/state` API endpoint. The node will periodically update this struct and store it int he DB. The api server will query it from the DB to serve it. - config: historydb.NodeConfig in JSON. This struct contains node configuration parameters that the API needs to be aware of. It's updated once every time the node starts. - constants: historydb.Constants in JSON. This struct contains all the hermez network constants gathered via the ethereum client by the node. It's written once every time the node starts. - The HistoryDB contains methods to get and update each one of these columns individually. - The HistoryDB contains all methods that query the DB and prepare objects that will appear in the StateAPI endpoint. - The configuration used in for the `serveapi` cli/node command is defined in `config.APIServer`, and is a subset of `node.Config` in order to allow reusing the same configuration file of the node if desired. - A new object is introduced in the api: `StateAPIUpdater`, which contains all the necessary information to update the StateAPI in the DB periodically by the node. - Moved the types `SCConsts`, `SCVariables` and `SCVariablesPtr` from `syncrhonizer` to `common` for convenience.
3 years ago
Allow serving API only via new cli command - Add new command to the cli/node: `serveapi` that alows serving the API just by connecting to the PostgreSQL database. The mode flag should me passed in order to select whether we are connecting to a synchronizer database or a coordinator database. If `coord` is chosen as mode, the coordinator endpoints can be activated in order to allow inserting l2txs and authorizations into the L2DB. Summary of the implementation details - New SQL table with 3 columns (plus `item_id` pk). The table only contains a single row with `item_id` = 1. Columns: - state: historydb.StateAPI in JSON. This is the struct that is served via the `/state` API endpoint. The node will periodically update this struct and store it int he DB. The api server will query it from the DB to serve it. - config: historydb.NodeConfig in JSON. This struct contains node configuration parameters that the API needs to be aware of. It's updated once every time the node starts. - constants: historydb.Constants in JSON. This struct contains all the hermez network constants gathered via the ethereum client by the node. It's written once every time the node starts. - The HistoryDB contains methods to get and update each one of these columns individually. - The HistoryDB contains all methods that query the DB and prepare objects that will appear in the StateAPI endpoint. - The configuration used in for the `serveapi` cli/node command is defined in `config.APIServer`, and is a subset of `node.Config` in order to allow reusing the same configuration file of the node if desired. - A new object is introduced in the api: `StateAPIUpdater`, which contains all the necessary information to update the StateAPI in the DB periodically by the node. - Moved the types `SCConsts`, `SCVariables` and `SCVariablesPtr` from `syncrhonizer` to `common` for convenience.
3 years ago
Update coordinator, call all api update functions - Common: - Rename Block.EthBlockNum to Block.Num to avoid unneeded repetition - API: - Add UpdateNetworkInfoBlock to update just block information, to be used when the node is not yet synchronized - Node: - Call API.UpdateMetrics and UpdateRecommendedFee in a loop, with configurable time intervals - Synchronizer: - When mapping events by TxHash, use an array to support the possibility of multiple calls of the same function happening in the same transaction (for example, a smart contract in a single transaction could call withdraw with delay twice, which would generate 2 withdraw events, and 2 deposit events). - In Stats, keep entire LastBlock instead of just the blockNum - In Stats, add lastL1BatchBlock - Test Stats and SCVars - Coordinator: - Enable writing the BatchInfo in every step of the pipeline to disk (with JSON text files) for debugging purposes. - Move the Pipeline functionality from the Coordinator to its own struct (Pipeline) - Implement shouldL1lL2Batch - In TxManager, implement logic to perform several attempts when doing ethereum node RPC calls before considering the error. (Both for calls to forgeBatch and transaction receipt) - In TxManager, reorganize the flow and note the specific points in which actions are made when err != nil - HistoryDB: - Implement GetLastL1BatchBlockNum: returns the blockNum of the latest forged l1Batch, to help the coordinator decide when to forge an L1Batch. - EthereumClient and test.Client: - Update EthBlockByNumber to return the last block when the passed number is -1.
3 years ago
  1. package api
  2. import (
  3. "fmt"
  4. "strconv"
  5. "testing"
  6. ethCommon "github.com/ethereum/go-ethereum/common"
  7. "github.com/hermeznetwork/hermez-node/common"
  8. "github.com/hermeznetwork/hermez-node/db/historydb"
  9. "github.com/mitchellh/copystructure"
  10. "github.com/stretchr/testify/assert"
  11. )
  12. type testSlot struct {
  13. ItemID uint64 `json:"itemId"`
  14. SlotNum int64 `json:"slotNum"`
  15. FirstBlock int64 `json:"firstBlock"`
  16. LastBlock int64 `json:"lastBlock"`
  17. OpenAuction bool `json:"openAuction"`
  18. WinnerBid *testBid `json:"bestBid"`
  19. }
  20. type testSlotsResponse struct {
  21. Slots []testSlot `json:"slots"`
  22. PendingItems uint64 `json:"pendingItems"`
  23. }
  24. func (t testSlotsResponse) GetPending() (pendingItems, lastItemID uint64) {
  25. pendingItems = t.PendingItems
  26. lastItemID = t.Slots[len(t.Slots)-1].ItemID
  27. return pendingItems, lastItemID
  28. }
  29. func (t testSlotsResponse) Len() int {
  30. return len(t.Slots)
  31. }
  32. func (t testSlotsResponse) New() Pendinger { return &testSlotsResponse{} }
  33. func (a *API) genTestSlots(nSlots int, lastBlockNum int64, bids []testBid, auctionVars common.AuctionVariables) []testSlot {
  34. tSlots := []testSlot{}
  35. bestBids := make(map[int64]testBid)
  36. // It's assumed that bids for each slot will be received in increasing order
  37. for i := range bids {
  38. bestBids[bids[i].SlotNum] = bids[i]
  39. }
  40. for i := int64(0); i < int64(nSlots); i++ {
  41. bid, ok := bestBids[i]
  42. firstBlock, lastBlock := a.getFirstLastBlock(int64(i))
  43. tSlot := testSlot{
  44. SlotNum: int64(i),
  45. FirstBlock: firstBlock,
  46. LastBlock: lastBlock,
  47. OpenAuction: a.isOpenAuction(lastBlockNum, int64(i), auctionVars),
  48. }
  49. if ok {
  50. tSlot.WinnerBid = &bid
  51. }
  52. tSlots = append(tSlots, tSlot)
  53. }
  54. return tSlots
  55. }
  56. func (a *API) getEmptyTestSlot(slotNum, lastBlock int64, auctionVars common.AuctionVariables) testSlot {
  57. firstSlotBlock, lastSlotBlock := a.getFirstLastBlock(slotNum)
  58. slot := testSlot{
  59. SlotNum: slotNum,
  60. FirstBlock: firstSlotBlock,
  61. LastBlock: lastSlotBlock,
  62. OpenAuction: a.isOpenAuction(lastBlock, slotNum, auctionVars),
  63. WinnerBid: nil,
  64. }
  65. return slot
  66. }
  67. func TestGetSlot(t *testing.T) {
  68. endpoint := apiURL + "slots/"
  69. for _, slot := range tc.slots {
  70. fetchedSlot := testSlot{}
  71. assert.NoError(
  72. t, doGoodReq(
  73. "GET",
  74. endpoint+strconv.Itoa(int(slot.SlotNum)),
  75. nil, &fetchedSlot,
  76. ),
  77. )
  78. assertSlot(t, slot, fetchedSlot)
  79. }
  80. // Slot with WinnerBid == nil
  81. slotNum := int64(15)
  82. fetchedSlot := testSlot{}
  83. assert.NoError(
  84. t, doGoodReq(
  85. "GET",
  86. endpoint+strconv.Itoa(int(slotNum)),
  87. nil, &fetchedSlot,
  88. ),
  89. )
  90. // ni, err := api.h.GetNodeInfoAPI()
  91. // assert.NoError(t, err)
  92. emptySlot := api.getEmptyTestSlot(slotNum, 0, tc.auctionVars)
  93. assertSlot(t, emptySlot, fetchedSlot)
  94. // Invalid slotNum
  95. path := endpoint + strconv.Itoa(-2)
  96. err := doBadReq("GET", path, nil, 400)
  97. assert.NoError(t, err)
  98. }
  99. func TestGetSlots(t *testing.T) {
  100. endpoint := apiURL + "slots"
  101. fetchedSlots := []testSlot{}
  102. appendIter := func(intr interface{}) {
  103. for i := 0; i < len(intr.(*testSlotsResponse).Slots); i++ {
  104. tmp, err := copystructure.Copy(intr.(*testSlotsResponse).Slots[i])
  105. if err != nil {
  106. panic(err)
  107. }
  108. fetchedSlots = append(fetchedSlots, tmp.(testSlot))
  109. }
  110. }
  111. // All slots with maxSlotNum filter
  112. maxSlotNum := tc.slots[len(tc.slots)-1].SlotNum + 5
  113. limit := 1
  114. path := fmt.Sprintf("%s?maxSlotNum=%d&limit=%d", endpoint, maxSlotNum, limit)
  115. err := doGoodReqPaginated(path, historydb.OrderAsc, &testSlotsResponse{}, appendIter)
  116. assert.NoError(t, err)
  117. allSlots := tc.slots
  118. // ni, err := api.h.GetNodeInfoAPI()
  119. // assert.NoError(t, err)
  120. for i := tc.slots[len(tc.slots)-1].SlotNum; i < maxSlotNum; i++ {
  121. emptySlot := api.getEmptyTestSlot(i+1, 0, tc.auctionVars)
  122. allSlots = append(allSlots, emptySlot)
  123. }
  124. assertSlots(t, allSlots, fetchedSlots)
  125. // All slots with maxSlotNum filter, in reverse order
  126. fetchedSlots = []testSlot{}
  127. limit = 3
  128. path = fmt.Sprintf("%s?maxSlotNum=%d&limit=%d", endpoint, maxSlotNum, limit)
  129. err = doGoodReqPaginated(path, historydb.OrderDesc, &testSlotsResponse{}, appendIter)
  130. assert.NoError(t, err)
  131. flippedAllSlots := []testSlot{}
  132. for i := len(allSlots) - 1; i >= 0; i-- {
  133. flippedAllSlots = append(flippedAllSlots, allSlots[i])
  134. }
  135. assertSlots(t, flippedAllSlots, fetchedSlots)
  136. // maxSlotNum & wonByEthereumAddress
  137. fetchedSlots = []testSlot{}
  138. limit = 1
  139. var bidderAddr ethCommon.Address
  140. for i := 0; i < len(tc.slots); i++ {
  141. if tc.slots[i].WinnerBid != nil {
  142. bidderAddr = tc.slots[i].WinnerBid.Bidder
  143. }
  144. }
  145. path = fmt.Sprintf("%s?maxSlotNum=%d&wonByEthereumAddress=%s&limit=%d", endpoint, maxSlotNum, bidderAddr.String(), limit)
  146. err = doGoodReqPaginated(path, historydb.OrderAsc, &testSlotsResponse{}, appendIter)
  147. assert.NoError(t, err)
  148. bidderAddressSlots := []testSlot{}
  149. for i := 0; i < len(tc.slots); i++ {
  150. if tc.slots[i].WinnerBid != nil {
  151. if tc.slots[i].WinnerBid.Bidder == bidderAddr {
  152. bidderAddressSlots = append(bidderAddressSlots, tc.slots[i])
  153. }
  154. }
  155. }
  156. assertSlots(t, bidderAddressSlots, fetchedSlots)
  157. // maxSlotNum & wonByEthereumAddress, in reverse order
  158. fetchedSlots = []testSlot{}
  159. limit = 1
  160. path = fmt.Sprintf("%s?maxSlotNum=%d&wonByEthereumAddress=%s&limit=%d", endpoint, maxSlotNum, bidderAddr.String(), limit)
  161. err = doGoodReqPaginated(path, historydb.OrderDesc, &testSlotsResponse{}, appendIter)
  162. assert.NoError(t, err)
  163. flippedBidderAddressSlots := []testSlot{}
  164. for i := len(bidderAddressSlots) - 1; i >= 0; i-- {
  165. flippedBidderAddressSlots = append(flippedBidderAddressSlots, bidderAddressSlots[i])
  166. }
  167. assertSlots(t, flippedBidderAddressSlots, fetchedSlots)
  168. // finishedAuction
  169. fetchedSlots = []testSlot{}
  170. limit = 15
  171. path = fmt.Sprintf("%s?finishedAuction=%t&limit=%d", endpoint, true, limit)
  172. err = doGoodReqPaginated(path, historydb.OrderAsc, &testSlotsResponse{}, appendIter)
  173. assert.NoError(t, err)
  174. currentSlot := api.getCurrentSlot(tc.blocks[len(tc.blocks)-1].Num)
  175. finishedAuctionSlots := []testSlot{}
  176. for i := 0; i < len(tc.slots); i++ {
  177. finishAuction := currentSlot + int64(tc.auctionVars.ClosedAuctionSlots)
  178. if tc.slots[i].SlotNum <= finishAuction {
  179. finishedAuctionSlots = append(finishedAuctionSlots, tc.slots[i])
  180. } else {
  181. break
  182. }
  183. }
  184. assertSlots(t, finishedAuctionSlots, fetchedSlots)
  185. //minSlot + maxSlot
  186. limit = 10
  187. minSlotNum := tc.slots[3].SlotNum
  188. maxSlotNum = tc.slots[len(tc.slots)-1].SlotNum - 1
  189. fetchedSlots = []testSlot{}
  190. path = fmt.Sprintf("%s?maxSlotNum=%d&minSlotNum=%d&limit=%d", endpoint, maxSlotNum, minSlotNum, limit)
  191. err = doGoodReqPaginated(path, historydb.OrderAsc, &testSlotsResponse{}, appendIter)
  192. assert.NoError(t, err)
  193. minMaxBatchNumSlots := []testSlot{}
  194. for i := 0; i < len(tc.slots); i++ {
  195. if tc.slots[i].SlotNum >= minSlotNum && tc.slots[i].SlotNum <= maxSlotNum {
  196. minMaxBatchNumSlots = append(minMaxBatchNumSlots, tc.slots[i])
  197. }
  198. }
  199. assertSlots(t, minMaxBatchNumSlots, fetchedSlots)
  200. //minSlot + maxSlot
  201. limit = 15
  202. minSlotNum = tc.slots[0].SlotNum
  203. maxSlotNum = tc.slots[0].SlotNum
  204. fetchedSlots = []testSlot{}
  205. path = fmt.Sprintf("%s?maxSlotNum=%d&minSlotNum=%d&limit=%d", endpoint, maxSlotNum, minSlotNum, limit)
  206. err = doGoodReqPaginated(path, historydb.OrderAsc, &testSlotsResponse{}, appendIter)
  207. assert.NoError(t, err)
  208. minMaxBatchNumSlots = []testSlot{}
  209. for i := 0; i < len(tc.slots); i++ {
  210. if tc.slots[i].SlotNum >= minSlotNum && tc.slots[i].SlotNum <= maxSlotNum {
  211. minMaxBatchNumSlots = append(minMaxBatchNumSlots, tc.slots[i])
  212. }
  213. }
  214. assertSlots(t, minMaxBatchNumSlots, fetchedSlots)
  215. // Only empty Slots
  216. limit = 2
  217. minSlotNum = tc.slots[len(tc.slots)-1].SlotNum + 1
  218. maxSlotNum = tc.slots[len(tc.slots)-1].SlotNum + 5
  219. fetchedSlots = []testSlot{}
  220. path = fmt.Sprintf("%s?maxSlotNum=%d&minSlotNum=%d&limit=%d", endpoint, maxSlotNum, minSlotNum, limit)
  221. err = doGoodReqPaginated(path, historydb.OrderAsc, &testSlotsResponse{}, appendIter)
  222. assert.NoError(t, err)
  223. emptySlots := []testSlot{}
  224. for i := 0; i < len(allSlots); i++ {
  225. if allSlots[i].SlotNum >= minSlotNum && allSlots[i].SlotNum <= maxSlotNum {
  226. emptySlots = append(emptySlots, allSlots[i])
  227. }
  228. }
  229. assertSlots(t, emptySlots, fetchedSlots)
  230. // Only empty Slots, in reverse order
  231. limit = 4
  232. minSlotNum = tc.slots[len(tc.slots)-1].SlotNum + 1
  233. maxSlotNum = tc.slots[len(tc.slots)-1].SlotNum + 5
  234. fetchedSlots = []testSlot{}
  235. path = fmt.Sprintf("%s?maxSlotNum=%d&minSlotNum=%d&limit=%d", endpoint, maxSlotNum, minSlotNum, limit)
  236. err = doGoodReqPaginated(path, historydb.OrderDesc, &testSlotsResponse{}, appendIter)
  237. assert.NoError(t, err)
  238. flippedEmptySlots := []testSlot{}
  239. for i := 0; i < len(flippedAllSlots); i++ {
  240. if flippedAllSlots[i].SlotNum >= minSlotNum && flippedAllSlots[i].SlotNum <= maxSlotNum {
  241. flippedEmptySlots = append(flippedEmptySlots, flippedAllSlots[i])
  242. }
  243. }
  244. assertSlots(t, flippedEmptySlots, fetchedSlots)
  245. // 400
  246. // No filters
  247. path = fmt.Sprintf("%s?limit=%d", endpoint, limit)
  248. err = doBadReq("GET", path, nil, 400)
  249. assert.NoError(t, err)
  250. // Invalid maxSlotNum
  251. path = fmt.Sprintf("%s?maxSlotNum=%d", endpoint, -2)
  252. err = doBadReq("GET", path, nil, 400)
  253. assert.NoError(t, err)
  254. // Invalid wonByEthereumAddress
  255. path = fmt.Sprintf("%s?maxSlotNum=%d&wonByEthereumAddress=%s", endpoint, maxSlotNum, "0xG0000001")
  256. err = doBadReq("GET", path, nil, 400)
  257. assert.NoError(t, err)
  258. // Invalid minSlotNum / maxSlotNum (minSlotNum > maxSlotNum)
  259. maxSlotNum = tc.slots[1].SlotNum
  260. minSlotNum = tc.slots[4].SlotNum
  261. path = fmt.Sprintf("%s?maxSlotNum=%d&minSlotNum=%d&limit=%d", endpoint, maxSlotNum, minSlotNum, limit)
  262. err = doBadReq("GET", path, nil, 400)
  263. assert.NoError(t, err)
  264. // 404
  265. maxSlotNum = tc.slots[1].SlotNum
  266. path = fmt.Sprintf("%s?maxSlotNum=%d&wonByEthereumAddress=%s&limit=%d", endpoint, maxSlotNum, tc.coordinators[3].Bidder.String(), limit)
  267. err = doBadReq("GET", path, nil, 404)
  268. assert.NoError(t, err)
  269. }
  270. func assertSlots(t *testing.T, expected, actual []testSlot) {
  271. assert.Equal(t, len(expected), len(actual))
  272. for i := 0; i < len(expected); i++ {
  273. assertSlot(t, expected[i], actual[i])
  274. }
  275. }
  276. func assertSlot(t *testing.T, expected, actual testSlot) {
  277. if actual.WinnerBid != nil {
  278. assert.Equal(t, expected.WinnerBid.Timestamp.Unix(), actual.WinnerBid.Timestamp.Unix())
  279. expected.WinnerBid.Timestamp = actual.WinnerBid.Timestamp
  280. actual.WinnerBid.ItemID = expected.WinnerBid.ItemID
  281. }
  282. actual.ItemID = expected.ItemID
  283. assert.Equal(t, expected, actual)
  284. }