BlockStack : How can i setup a blockstack full node (not testnet)

devd · July 15, 2021, 7:30am

@jwiley Thanks for the detailed explanation on my previous query , makes complete sense to use the API + Stack local node + BTC Node.

Before that i was just trying to understand the json structure of couple of queries and map it with explorer to understand the the operations where STACKS is involved.

My requirement is to get the balance and transactions for stacks address at a particular blockheight:

Balance at a blockheight to get balance - Did not get any Apis to get this information . Could you please validate.
Transactions till a blockHeight - Did not get any direct Apis but got some Apis which gets all the transactions and from there we need to filter , but when i validate this with stacks explorer not able to map the json data . Could you please help me understand how can i use this data .

Sample address - SP3DQR29HQYTGGA5287NJPM3V1WC6J0HSPXQHFJKB
In explorer - Total 15 transactions
Avialable Balance - 1.129834

using this Api - /v1/accounts/SP3DQR29HQYTGGA5287NJPM3V1WC6J0HSPXQHFJKB/history?page=0

against my node and also
https://core.blockstack.org/v1/accounts/SP3DQR29HQYTGGA5287NJPM3V1WC6J0HSPXQHFJKB/history?page=0

and

for page 1 as well

getting the json structure where credits and debits are involved and a BTC address which i am not able to relate with the stacks explorer -

Can you please help me to understand the transactions and when we do the calculations it should reach to the Stacks balance what ever shown in explorer.

i have not provided the json structure over here but you can run the Apis provided above to see the structure.

jwiley · July 15, 2021, 2:18pm

This is what you’re looking for: Stacks 2.0 Blockchain API

the core.blockstack.org endpoints you’ve referenced are deprecated, any stacks blocks after bitcoin block 666053 won’t be available from that endpoint.

the explorer and the above api have all the data you’re looking for - but i will caution not to spam the those endpoints, since they are public resources we have a moderate rate-limit enforced to keep the services running.

if you need to pull a lot of data, or crawl the data - i’d encourage you to run your own API node as described in my other replies.
otherwise, depending on the volume of requests you want to make - you may see your requests throttled after a while.

now to your question about calculations - the explorer uses an API service (again, linked above already) to display the data in the webpage. the stacks-node will receive the block data initially, then send that event to the API server which then stores it in a postgres DB. the explorer will then make requests for data from that API using the endpoints described in the link above in this reply: Stacks 2.0 Blockchain API

and to be clear, all the components i’ve described here can be run by anyone - the source code is all readily available.

devd · July 15, 2021, 4:34pm

Thanks @jwiley I have just given the core.blockstack api as a reference as i was getting same kind of response from my own node set up (blockstack + bitcoin node )
Below is the query which i am using in my node set up:

sudo curl -v http:///v1/accounts/SP3DQR29HQYTGGA5287NJPM3V1WC6J0HSPXQHFJKB/history?page=0

So as per you response its deprecated and i should go by the Api + Blovkstack Node + Btc Node and then try the queries from “Stacks 2.0 Blockchain API” and thats where i will be able to match with the explorer data .

Is my understanding correct.

jwiley · July 15, 2021, 4:54pm

yes and no - it all depends on what type of data you’re looking to retrieve and how often.
there is an RPC endpoint in the stacks-blockchain, but be aware these are blocking requests.

github.com

blockstack/stacks-blockchain/blob/master/docs/rpc-endpoints.md

# RPC Endpoints

### POST /v2/transactions

This endpoint is for posting _raw_ transaction data to the node's mempool.

Rejections result in a 400 error, with JSON data in the form:

```
{
  "error": "transaction rejected",
  "reason": "BadNonce",
  "reason_data": {
    "actual": 3,
    "expected": 0,
    "is_origin": true,
    "principal": "ST2MVNFYF6H9DCMAV3HVNHTJVVE3CFWT1JYMH1EZB"
  },
  "txid": "0x4068179cb9169b969c80518d83890f8b808a70ab998dd227149221be9480a616"
}

This file has been truncated. show original

github.com

blockstack/stacks-blockchain/blob/master/docs/rpc/openapi.yaml

openapi: 3.0.2
servers:
  - url: http://localhost:20443/
    description: Local
info:
  title: Stacks 2.0 RPC API
  version: '1.0.0'
  description: |
    This is the documentation for the `stacks-node` RPC interface.

paths:
  /v2/transactions:
    post:
      summary: Broadcast raw transaction
      tags:
        - Transactions
      description: Broadcast raw transactions on the network. You can use the [@stacks/transactions](https://github.com/blockstack/stacks.js) project to generate a raw transaction payload.
      operationId: post_core_node_transactions
      requestBody:
        content:

This file has been truncated. show original

github.com

blockstack/stacks-blockchain/blob/master/src/net/http.rs#L91-L142


lazy_static! {
    static ref PATH_GETINFO: Regex = Regex::new(r#"^/v2/info$"#).unwrap();
    static ref PATH_GETPOXINFO: Regex = Regex::new(r#"^/v2/pox$"#).unwrap();
    static ref PATH_GETNEIGHBORS: Regex = Regex::new(r#"^/v2/neighbors$"#).unwrap();
    static ref PATH_GETBLOCK: Regex = Regex::new(r#"^/v2/blocks/([0-9a-f]{64})$"#).unwrap();
    static ref PATH_GETMICROBLOCKS_INDEXED: Regex =
        Regex::new(r#"^/v2/microblocks/([0-9a-f]{64})$"#).unwrap();
    static ref PATH_GETMICROBLOCKS_CONFIRMED: Regex =
        Regex::new(r#"^/v2/microblocks/confirmed/([0-9a-f]{64})$"#).unwrap();
    static ref PATH_GETMICROBLOCKS_UNCONFIRMED: Regex =
        Regex::new(r#"^/v2/microblocks/unconfirmed/([0-9a-f]{64})/([0-9]{1,5})$"#).unwrap();
    static ref PATH_GETTRANSACTION_UNCONFIRMED: Regex =
        Regex::new(r#"^/v2/transactions/unconfirmed/([0-9a-f]{64})$"#).unwrap();
    static ref PATH_POSTTRANSACTION: Regex = Regex::new(r#"^/v2/transactions$"#).unwrap();
    static ref PATH_POSTBLOCK: Regex = Regex::new(r#"^/v2/blocks/upload/([0-9a-f]{40})$"#).unwrap();
    static ref PATH_POSTMICROBLOCK: Regex = Regex::new(r#"^/v2/microblocks$"#).unwrap();
    static ref PATH_GET_ACCOUNT: Regex = Regex::new(&format!(
        "^/v2/accounts/(?P<principal>{})$",
        *PRINCIPAL_DATA_REGEX
    ))

This file has been truncated. show original

using that as a reference, a similar query to your curl would look like this:

curl -sL http://<local stacks node>:20443/v2/accounts/SP3DQR29HQYTGGA5287NJPM3V1WC6J0HSPXQHFJKB?page=0 | jq

I do want to stress as well that not all API endpoints you’re looking at for v1 are ported to v2 - some have been ported, some are not, and some are slightly different. they’re all described in the above links.

vishalgupta96 · September 6, 2021, 3:15pm

Hi Again,

I have followed the exact steps mentioned on the below links:
stacks-blockchain-api/running_an_api.md at master · blockstack/stacks-blockchain-api · GitHub

My all the 3 nodes in the docker containers are up and running and stacks blockchain is syncing fine. But when i am verifying the node , my CURL query to stacks-blockchain node works fine and giving the expected response but query to API node is not giving a response
curl -sL localhost:20443/v2/info (works fine)
but below query does not work from the same server on which it is running:
curl -L http://localhost:3999/v2/info (Recv failure: Connection reset by peer)

I have followed the exact same process and config files provided as it is. Please help.

jwiley · September 7, 2021, 12:59pm

Without seeing the logs, there could be many reasons the port is not opening for the API.
You’ll need to share some of the API’s logs so we can find what’s happening there, but I would guess it’s misconfigured since port 3999 opens after it can communicate with the stacks-blockchain on port 20443

jwiley · September 8, 2021, 5:29pm

@vishalgupta96 any new info you can provide here?
particualry i’m looking for command output from the above curl’s you ran, and the API logs

vishalgupta96 · October 8, 2021, 6:59am

@jwiley : Hi Jesse, we have used the above URL for running 3 nodes (API, Stacks & Postgres) and also used the below link for standalone node: https://github.com/blockstack/stacks-blockchain-api
and in the both type of setup i am getting below docker logs and i am unable to query the node:

used below docker run command to start the standalone node:
docker run -p 3999:3999 blockstack/stacks-blockchain-api-standalone mainnet

On running this, i am getting below logs in node:
INFO [1633675537.925778] [testnet/stacks-node/src/run_loop/neon.rs:428] [main] Downloaded burnchain blocks up to height 704051; new target height is 704051; next_height = 704051, block_height = 704051 INFO [1633675537.976387] [testnet/stacks-node/src/run_loop/neon.rs:501] [main] Synchronized full burnchain up to height 704051. Proceeding to mine blocks INFO [1633675543.645034] [src/net/http.rs:1571] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 127.0.0.1:60682, path: /v2/info, query: WARN [1633675543.712766] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675543.712818] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675543.712844] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675543.712869] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675543.712894] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675543.712919] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) INFO [1633675549.113535] [testnet/stacks-node/src/run_loop/neon.rs:428] [main] Downloaded burnchain blocks up to height 704051; new target height is 704051; next_height = 704051, block_height = 704051 INFO [1633675549.146951] [testnet/stacks-node/src/run_loop/neon.rs:501] [main] Synchronized full burnchain up to height 704051. Proceeding to mine blocks INFO [1633675554.133496] [src/net/http.rs:1571] [p2p] Handle HTTPRequest, verb: GET, peer_addr: 127.0.0.1:60720, path: /v2/info, query: WARN [1633675554.203447] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675554.203501] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675554.203527] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675554.203554] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675554.203578] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) WARN [1633675554.203602] [src/net/connection.rs:559] [p2p] Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Failed to parse HTTP request: Version"); response error: DeserializeError("Unsupported HTTP content type")) INFO [1633675560.484671] [testnet/stacks-node/src/run_loop/neon.rs:428] [main] Downloaded burnchain blocks up to height 704051; new target height is 704051; next_height = 704051, block_height = 704051 INFO [1633675560.518253] [testnet/stacks-node/src/run_loop/neon.rs:501] [main] Synchronized full burnchain up to height 704051. Proceeding to mine blocks

But i am unable to query the node and unable to use any APIs. For ex:
Request:
sudo curl -L localhost:3999/v2/info

Response:
curl: (56) Recv failure: Connection reset by peer

Could you please help here… what’s wrong with above?

jwiley · October 8, 2021, 1:58pm

Without debug logging, it’s impossible to say why you’re seeing those messages: https://github.com/blockstack/stacks-blockchain/blob/master/src/net/connection.rs#L557-L566

The messages themselves aren’t necessarily a problem - they’re simply saying they don’t understand the http request.
But, without debug logging we don’t know what the request is or even where it’s coming from.

Further, I wouldn’t recommend using the docker image you’re running - it’s mainly used for testing and I’m not sure how up to date it is.

If you’re interested in running an API node, I’d encourage you to look at the docs here: Running an API node | Stacks

Just be sure you have at least 4GB of free memory and ideally 2vcpu at a minimum.

vishalgupta96 · October 8, 2021, 2:33pm

@jwiley : Hi Jesse, As i have mentioned above - i have started API node by both ways. One is the steps provided the link provided by you and another is using “standalone” image. I understand about the “standalone” that it is only for testing but the other node which i have created using the above link only. But i am getting the same logs in that node also. Could you suggest from where i can get the debug logs to share it with you?

jwiley · October 8, 2021, 3:07pm

using the commands in the link above, you’d adjust it to look like this (notice the extra env vars that are added):

docker run -d --rm \
  --name stacks-blockchain \
  --net=stacks-blockchain \
  -e RUST_BACKTRACE=full \
  -e STACKS_LOG_DEBUG=1 \
  -v $(pwd)/persistent-data/stacks-blockchain:/root/stacks-node/data \
  -v $(pwd)/config:/src/stacks-node \
  -p 20443:20443 \
  -p 20444:20444 \
  blockstack/stacks-blockchain \
/bin/stacks-node start --config /src/stacks-node/Config.toml

Once you have it running until you reach the error messages, you can save the logs simply run a command a link this to save the logs:
docker logs stacks-blockchain > logs.txt 2>&1

To share the logs, it depends on the size - but you’ll likely want to compress the logs and use a public file-sharing service like google drive, dropbox etc. you can send me the link (or if the file is small enough for email) to jwiley at hiro dot so

Shalini · October 19, 2021, 8:09am

Just wanted to check if this issue is resolved? if not, please share the exact error message. You can try doing the telnet to these ports from the local.

vishalgupta96 · October 19, 2021, 4:41pm

Hi @jwiley : I have uploaded the logs on sharepoint and shared the link over email with you. You should have received email from sconnors at deloitte dot com. Your email id jwiley at hiro dot so has been provided access to access the link.

Please check and confirm once you get to see it.

jwiley · October 25, 2021, 5:19pm

@vishalgupta96 sorry for the delayed response, I was off the internet for the past week.
I do see the emails you mentioned, I’ll take a look in a bit (hopefully they still are working) and get back to you.
If the links are no longer valid, i’ll let you know and we can try again now that i’m available.

jwiley · October 27, 2021, 3:18pm

@vishalgupta96 I have the logs and I am looking through them. I’ll get back to you soon

jwiley · October 27, 2021, 5:34pm

@vishalgupta96 can you share the configs you’re using that produced this logfile?
obviously remove any burnchain password if needed - you can send them to my email like you did with the logs. txt attachment is fine for these, they should be small.
I’d also like to see the .env file, or env vars you’re using for the API.

vishalgupta96 · October 28, 2021, 12:59pm

hi @jwiley, i have sent you the config file. However i could not find .env file. We have used the environment variables, you mentioned in above command , copying below:

docker run -d --rm
–name stacks-blockchain
–net=stacks-blockchain
-e RUST_BACKTRACE=full
-e STACKS_LOG_DEBUG=1
-v $(pwd)/persistent-data/stacks-blockchain:/root/stacks-node/data
-v $(pwd)/config:/src/stacks-node
-p 20443:20443
-p 20444:20444
blockstack/stacks-blockchain
/bin/stacks-node start --config /src/stacks-node/Config.toml

jwiley · October 28, 2021, 9:18pm

@vishalgupta96
based on the logs and the envvars you shared over email - it looks like the API env vars are simply incorrect for this type of setup.

github.com

blockstack/stacks-blockchain-api/blob/master/running_an_api.md#stacks-blockchain-api

# Running a stacks-blockchain API instance with Docker

In this document, we'll go over how to run a [stacks-blockchain-api](https://github.com/blockstack/stacks-blockchain-api) instance.  
There are several components involved here to have a working setup, and we'll go over each.  
We'll focus on the **easy** path to get the services running, which today is using Docker.  
Please note that the following guide is meant for a Unix-like OS (Linux/MacOS) - the commands *may* work on Windows but will likely need some adjustments (PR's welcome!).

- [Running a stacks-blockchain API instance with Docker](#running-a-stacks-blockchain-api-instance-with-docker)
  - [Requirements](#requirements)
    - [Important](#important)
    - [Firewalling](#firewalling)
    - [Initial Setup](#initial-setup)
  - [Postgres](#postgres)
    - [Starting postgres](#starting-postgres)
    - [Stopping Postgres](#stopping-postgres)
  - [Stacks Blockchain API](#stacks-blockchain-api)
    - [Starting stacks-blockchain-api](#starting-stacks-blockchain-api)
    - [Stopping stacks-blockchain-api](#stopping-stacks-blockchain-api)
  - [Stacks Blockchain](#stacks-blockchain)
    - [Starting stacks-blockchain](#starting-stacks-blockchain)

This file has been truncated. show original

To use your previous docker command, you’d have to run the other containers on the same network stacks-blockchain - for a total of 3 containers.

stacks-blockchain
stacks-blockchain-api
postgres

I’d encourage you to use that .env file in the doc, or at least adjust the env vars as follows:

NODE_ENV=production
GIT_TAG=master
PG_HOST=postgres
PG_PORT=5432
PG_USER=postgres
PG_PASSWORD=postgres
PG_DATABASE=postgres
STACKS_CHAIN_ID=0x00000001
V2_POX_MIN_AMOUNT_USTX=90000000260
STACKS_CORE_EVENT_PORT=3700
STACKS_CORE_EVENT_HOST=0.0.0.0
STACKS_BLOCKCHAIN_API_PORT=3999
STACKS_BLOCKCHAIN_API_HOST=0.0.0.0
STACKS_BLOCKCHAIN_API_DB=pg
STACKS_CORE_RPC_HOST=stacks-blockchain
STACKS_CORE_RPC_PORT=20443
BNS_IMPORT_DIR=/bns-data

important to note here the following env vars:
STACKS_CORE_RPC_HOST=stacks-blockchain - this value has to match the name of the stacks-blockchain container
PG_HOST=postgres - this value has to match the name of the postgres container
BNS_IMPORT_DIR=/bns-data - this variable is entirely optional, if you enable it though the startup of the API will take quite a bit longer.

for the postgres user/pass - adjust as you need to, these are defaults for the postgres image mentioned in the doc above https://github.com/blockstack/stacks-blockchain-api/blob/master/running_an_api.md#postgres

github.com

blockstack/stacks-blockchain-api/blob/master/running_an_api.md#starting-stacks-blockchain-api

# Running a stacks-blockchain API instance with Docker

In this document, we'll go over how to run a [stacks-blockchain-api](https://github.com/blockstack/stacks-blockchain-api) instance.  
There are several components involved here to have a working setup, and we'll go over each.  
We'll focus on the **easy** path to get the services running, which today is using Docker.  
Please note that the following guide is meant for a Unix-like OS (Linux/MacOS) - the commands *may* work on Windows but will likely need some adjustments (PR's welcome!).

- [Running a stacks-blockchain API instance with Docker](#running-a-stacks-blockchain-api-instance-with-docker)
  - [Requirements](#requirements)
    - [Important](#important)
    - [Firewalling](#firewalling)
    - [Initial Setup](#initial-setup)
  - [Postgres](#postgres)
    - [Starting postgres](#starting-postgres)
    - [Stopping Postgres](#stopping-postgres)
  - [Stacks Blockchain API](#stacks-blockchain-api)
    - [Starting stacks-blockchain-api](#starting-stacks-blockchain-api)
    - [Stopping stacks-blockchain-api](#stopping-stacks-blockchain-api)
  - [Stacks Blockchain](#stacks-blockchain)
    - [Starting stacks-blockchain](#starting-stacks-blockchain)

This file has been truncated. show original

docker run -d --rm \
    --name stacks-blockchain-api \
    --net=stacks-blockchain \
    --env-file $(pwd)/.env \
    -v $(pwd)/bns:/bns-data \
    -p 3700:3700 \
    -p 3999:3999 \
    blockstack/stacks-blockchain-api

notice the env-file option above - this is what you might be missing in your setup. 127.0.0.1 is the default.

One more thing to add - it’s better at this point to delete any data dirs you have in $(pwd)/persistent-data/stacks-blockchain.

sawru · November 2, 2021, 12:58am

Hello jWiley, I’ve taken over from Vishal on this. And I’ve set up the 3 containers (postgres, stacks-blockchain-api, and stacks-blockchain).

This query runs fine curl -sL localhost:20443/v2/info
This one is still problematic curl -sL localhost:3999/v2/info

failing with same error: curl: (56) Recv failure: Connection reset by peer

I’ve confirmed that my env-file is consistent with your above suggestions and the instructions here.

Was there anything else in the logs that could be the issue? Thanks - also since this thread has been going on for about a year, not sure if it would be easier to hop on a call to discuss any other solutions.

jwiley · November 2, 2021, 2:00pm

I don’t doubt that your .env file is correct - but based on what was shared with me, the env vars are not being used by the API container - that’s the root of the problem here.

I’m open to call - send me a note to the email a few comments above and we can work out a time.