Bitcoin P2P API getBlocks always returns 500 inv items

I’m trying to build a mini full node so that I can understand exactly how they operate. I’ve managed to get a few parts of the P2P api working however I am stuck with basic things like ‘getblocks’ and ‘getheaders’

For example I have the following block hashes I am interested in:

block_hash = "000000000000592589e55cda6e8a093998e8356ea770d4aaeb7c0f5439b147d7"
block_stop = "000000000000017a09017d52db538d7a9ddcc48311866d7e5fdbbbec7d0faad5"

Command: getblocks
Request: 0b110907676574626c6f636b73000000450000002a0af9950100000001000000000000592589e55cda6e8a093998e8356ea770d4aaeb7c0f5439b147d7000000000000017a09017d52db538d7a9ddcc48311866d7e5fdbbbec7d0faad5

My question is: Why am I getting back 500 inv items when the stop hash is only 1 block away from the block locator ? (Btw the same thing happens to me when I’m using getheaders instead of getblocks)

EDIT

Ok so I have swapped the endian-ness of the data I’m sending. I also had to swap the block locator and stop around in order to get a response (so the stop hash is at a block height lower than the locator) which makes me feel like I’m making more mistakes here.

When doing this I still receive 500 inv on the example above, however if I use these two instead:

block_stop = "0000000000000049a7ce260de7fcf63f34acf3fe1c5b2040ae77b9bdaf1abb38"
block_hash = "000000000000013d8a77f222805b1dca31d0fb48d60b29098c77609de5f93811

ACTUAL MESSAGE: 0b110907676574626c6f636b7300000045000000a83c12f901000000011138f9e59d60778c09290bd648fbd031ca1d5b8022f2778a3d0100000000000038bb1aafbdb977ae40205b1cfef3ac343ff6fce70d26cea74900000000000000

it correctly returns from the height I require, however it returns all of the blocks until the latest block instead of the one that I’m after. Am I misunderstanding the point of this API ? I was under the impression it returns block details but instead it just returns missing block hashes. How do I then find out what transactions happened in that block?

What benefits does Bitcoin have that other cryptocurrencies do not?

I have heard several times Bitcoin being called “the king of crypto.” It makes sense when you look at how the coin surpasses all the altcoins in terms of market capitalization and trading volume. No doubt that BTC is dominant in the crypto market.

Why? Why do people prefer to buy Bitcoin over other cryptocurrencies? If you disregard the sheer popularity of the coin, I don’t see what other value it has and why there is such a high demand.

The most notable argument is that people buy BTC for international money transfer and buying/selling. However, it seems that BTC would not come close as a perfect candidate for that. Stable coins (ex. Tether) having less volatility would be much better for such transactions. It seems pretty obvious that people and stores buying and selling in stablecoin would be so much better. Using such alternative can provide trust for customers and sellers that what they thought they paid or accepted will actually be what they thought of the upcoming day.

There are a lot of perks for using cryptocurrency. Transaction fees are minimal and transfer is speedy. There is potential for international money transfer, and it is not really tied down to massive regulation and manipulation. However, I’m just not finding the perks of why BTC is the best coin to do all these things.

If Libra is introduced as a stablecoin with a company like Facebook supporting it, why would there be a demand for BTC? Why is there still demand today when Tether can serve as a stable medium of transaction?

Ultimately, is BTC mostly only bought for investment/speculation or is there a real use for using this currency today where there are so many other altcoins to choose from?

Does a trusted blockchain import from BitcoinCore require that chainstate and blocks be from the same source?

Can the chainstate folder from a pruned node be used with the blocks folder from a full node to provide the full blockchain? Assume that both nodes have been running for months.

If the blocks folder contains references to filenames in the chainstate folder, then the answer is no. I know this because the filenames in chainstate on the full node are different from those on the pruned node.

If, on the other hand, the chainstate folder stands as a self-contained and full record of the UTXO Set including all pointers, filenames, and identifiers required to interpret it, then it seems that I will NOT need to import chainstate from the full node to the cloud server.

Bonus question! Out of curiosity, there’s another question here that doesn’t affect my situation, but it would be nice to understand why the answer is no if it’s no. If the chainstate folder on a full node got corrupted, could the chainstate folder from a pruned node be dropped in to recover?

Clarification

Both the blocks and chainstate folders reflect a block tip, and if the two folders reflect different block tips, they will be incompatible. I should have made it explicit that the question assumes that the two different sources of folders would both be synced up to the same block height.

An interesting edge case is what happens if they are using the same block height, but the blocks they used at that height are two valid but different blocks.

BIP118 (SIGHASH_NOINPUT) vs. ANYPREVOUT

I understand the motivation for SIGHASH_NOINPUT – it would help with offchain payment channels. It was proposed already in the Lightning paper but Eltoo will effectively work just with something like that. But what exactly is the ANYPREVOUT BIP about? It seems like some extension to be used with taproot scripts. But I don’t seem to get why you need a special public key type and chaperone signatures. Can anyone explain this part in plain english please?

How does Bitcoin Core manage its peer nodes? Is it possible to let it automatically switch to faster nodes?

It’s usually frustrating when my Bitcoin Core full node is syncing, especially just catching up from merely several hours/days ago.

I wonder how does Bitcoin Core manage its peers? Will it disconnect with slower nodes, and then turn to faster ones?

I once heard of the eclipse attack issue. However, as far as I know, Bitcoin Core syncs the block headers first, then it will always download full block data. It doesn’t seem to hurt if Bitcoin Core could behave in such way (switching to faster nodes) when it’s downloading blocks.

EDIT: According to my rough observation, fast nodes often seemed to be “stuck”, as long as slow peers were still not disconnected, in other words, manually disconnecting slow peers often seemed to “accelerate” the syncing process.

Is that merely some sort of illusion, or a real phenomenon? Will manually disconnecting nodes introduce any risk?

Auto load all wallets when start up

Every time I restart my bitcoind, all the wallets will need to be reloaded:

$ bitcoin-cli listwallets
[
  ""
]
$ bitcoin-cli loadwallet test01
$ bitcoin-cli listwallets
[
  "",
  "test01"
]

So I was wondering that is there any ways to auto load all the wallet under path /home/ubuntu/.bitcoin/testnet3/wallets when startup? I have tried adding some config in bitcoin.config, but all didn’t work for me:

wallet=/home/ubuntu/.bitcoin/testnet3/wallets
wallet=/home/ubuntu/.bitcoin/testnet3/wallets/test01
wallet=1

bitcoin core version: v0.18.1

bitcoind: cannot obtain a lock on wallet directory

Hello fellow bitcoiners,

I have a very annoying and mysterious issue with a bitcoind install. I had a blockchain corruption/stuck at some point. I deleted the old blockchain and copied an up-to-date blockchain from another machine. And now bitcoind does not succeed to run. When launched it complains about: “Cannot obtain a lock on wallet directory”. I have deleted the .lock file, bitcoind obviously not running. After relaunch, the file gets created again. ps -aux | grep bitcoin returns no running process. bitoind starts (as a systemd service, which wasn’t modified and run perfectly before, not in question). After the blockchain copy I chown the blockchain directory to the correct user on the target machine. I have enven tried chown 777 giving all possible rights on the blockchain directory. Still have the lock issue.

Here is the debug log:

2019-12-22T21:26:19Z Using data directory /usr/local/bitc/blockchain
2019-12-22T21:26:19Z Config file: /usr/local/bitc/blockchain/bitcoin.conf
2019-12-22T21:26:19Z Using at most 125 automatic connections (1024 file descriptors available)
2019-12-22T21:26:19Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2019-12-22T21:26:19Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2019-12-22T21:26:19Z Using 4 threads for script verification
2019-12-22T21:26:19Z scheduler thread start
2019-12-22T21:26:20Z HTTP: creating work queue of depth 16
2019-12-22T21:26:20Z No rpcpassword set - using random cookie authentication.
2019-12-22T21:26:20Z Generated RPC authentication cookie /usr/local/bitc/blockchain/.cookie
2019-12-22T21:26:20Z HTTP: starting 4 worker threads
2019-12-22T21:26:20Z Using wallet directory /usr/local/bitc/blockchain
2019-12-22T21:26:20Z init message: Verifying wallet(s)...
2019-12-22T21:26:20Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2019-12-22T21:26:20Z Using wallet /usr/local/bitc/blockchain
2019-12-22T21:26:20Z ERROR: Error while attempting to lock directory /usr/local/bitc/blockchain: Permission denied
2019-12-22T21:26:20Z Cannot obtain a lock on wallet directory /usr/local/bitc/blockchain. Another instance of bitcoin may be using it.
2019-12-22T21:26:20Z Error: Error initializing wallet database environment "/usr/local/bitc/blockchain"!
2019-12-22T21:26:20Z Shutdown: In progress...
2019-12-22T21:26:20Z scheduler thread interrupt
2019-12-22T21:26:20Z Shutdown: done

Does anyone has a clue?

Best regards, cr!ptal.

Check if multiple signatures match a specifc public key

Suppose you have a company which has three main leaders called A, B and C also the company have 5 shareholders called S1 to S5. The decision rule for spending any coin from company’s address is to either we present 1 of the main leaders signature and three other signature from shareholders.
For example following unlocking script works (since it contains one of the leaders):

<sig_S1> <sig_A> <sig_S2> <sig_S5> 

but not this one:

<sig_S1> <sig_S4> <sig_S3> <sig_S5> 

How I can check if amongst multi signatures there exist a signature which corresponds to a specific public key, in this case, I will learn that a leader’s signature surely exists. for example, if we find out from 5 signatures provided the signature of A exists, and the locking code will look like follows:

(3 times duplicate of all signatures provided)
(check for the existence of A in signatures)
OP_IF 
   4 <pub_A> <pub_S1> <pub_S2> ... <pub_S5> 5 OP_CHECKMULTISIG
OP_ENDIF
(the same as above for B)
(the same as above for C)

How do I calculate transaction fee for call to createpsbt()?

When my user invokes my app, he passes as an input parameter the desired value of the TX fee in satoshis per byte. I create a PSBT by calling createpsbt(). I exert complete control over the determination of inputs and outputs. For example, I don’t call walletcreatefundedpsbt(), because I don’t want bitcoind to select inputs on my behalf. I have a chicken and egg problem in that I need to call createpsbt() in order to determine the size of the TX in bytes, but I need to specify the fee amount before the call to createpsbt(). The best solution that I can think of is to execute a dummy call to createpsbt() using a dummy fee value, then measure the size of the TX, then do a real call to createpsbt() after correctly calculating the fee amount. Is there a better way?

Edit: I want to spend all UTXOs. I want two outputs, one to receive a fixed amount, the other to receive change less the fee. I have tried many incantations, I can’t get it to work. Here’s my latest attempt:

raw_psbt = rpc.walletcreatefundedpsbt(
    # all UTXOs, as returned by listunspent():
    inputs,
    # one recipient: [ { addr : amount } ]
    outputs,
    0,  # locktime
    {
        "replaceable" : True,
        # I would like for the change less the fee to go here:
        "changeAddress" : xxx,
        "includeWatching" : True,
        "feeRate" : 0.005,
    }
)

This fails with Signing transaction failed (-4). Any idea what I’m doing wrong?

Edit #2: I can’t use walletcreatefundedpsbt() as suggested below to create the PSBT, because that function requires that its inputs be “solvable”, which is not the case in my environment. More info at this link:

Errors with walletcreatefundedpsbt & non-solvable UTXOs

So my original question remains unresolved at this point.

Edit #3: So I asked on Slack. The problem is that our UTXOs are 1) not “solvable” and 2) cannot be expressed in the descriptor language, so Bitcoin Core has no way to calculate the TX size before the TX is signed. Apparently I would have to calculated it manually as explained in the attached screenshot.slack chat