实践Netty之消息长度编解码

Posted on | In | Visitors

message=[header1]+lengthfield+[header2]+body

A decoder that splits the received ByteBufs dynamically by the value of the length field in the message. It is particularly useful when you decode a binary message which has an integer header field that represents the length of the message body or the whole message. LengthFieldBasedFrameDecoder has many configuration parameters so that it can decode any message with a length field, which is often seen in proprietary client-server protocols. Here are some example that will give you the basic idea on which option does what.

2 bytes length field at offset 0, do not strip header The value of the length field in this example is 12 (0x0C) which represents the length of “HELLO, WORLD”. By default, the decoder assumes that the length field represents the number of the bytes that follows the length field. Therefore, it can be decoded with the simplistic parameter combination.

lengthFieldOffset = 0

lengthFieldLength = 2

lengthAdjustment = 0

initialBytesToStrip = 0 (= do not strip header)

BEFORE DECODE (14 bytes)   AFTER DECODE (14 bytes)  
Length Actual Content Length Actual Content
0x000C “HELLO, WORLD” 0x000C “HELLO, WORLD”

2 bytes length field at offset 0, strip header Because we can get the length of the content by calling ByteBuf.readableBytes(), you might want to strip the length field by specifying initialBytesToStrip. In this example, we specified 2, that is same with the length of the length field, to strip the first two bytes.

lengthFieldOffset = 0

lengthFieldLength = 2

lengthAdjustment = 0

initialBytesToStrip = 2 (= the length of the Length field)

BEFORE DECODE (14 bytes)   AFTER DECODE (12 bytes)  
Length Actual Content   Actual Content
0x000C “HELLO, WORLD”   “HELLO, WORLD”

2 bytes length field at offset 0, do not strip header, the length field represents the length of the whole message In most cases, the length field represents the length of the message body only, as shown in the previous examples. However, in some protocols, the length field represents the length of the whole message, including the message header. In such a case, we specify a non-zero lengthAdjustment. Because the length value in this example message is always greater than the body length by 2, we specify -2 as lengthAdjustment for compensation.

lengthFieldOffset = 0

lengthFieldLength = 2

lengthAdjustment = -2 (= the length of the Length field)

initialBytesToStrip = 0

BEFORE DECODE (14 bytes)   AFTER DECODE (12 bytes)  
Length Actual Content Length Actual Content
0x000E “HELLO, WORLD” 0x000E “HELLO, WORLD”

3 bytes length field at the end of 5 bytes header, do not strip header The following message is a simple variation of the first example. An extra header value is prepended to the message. lengthAdjustment is zero again because the decoder always takes the length of the prepended data into account during frame length calculation.

lengthFieldOffset = 2 (= the length of Header 1)

lengthFieldLength = 3

lengthAdjustment = 0

initialBytesToStrip = 0

BEFORE DECODE (17 bytes)     AFTER DECODE (17 bytes)    
Header 1 Length Actual Content Header 1 Length Actual Content
0xCAFE 0x00000C “HELLO, WORLD” 0xCAFE 0x00000C “HELLO, WORLD”

3 bytes length field at the beginning of 5 bytes header, do not strip header This is an advanced example that shows the case where there is an extra header between the length field and the message body. You have to specify a positive lengthAdjustment so that the decoder counts the extra header into the frame length calculation.

lengthFieldOffset = 0

lengthFieldLength = 3

lengthAdjustment = 2 (= the length of Header 1)

initialBytesToStrip = 0

BEFORE DECODE (17 bytes)     AFTER DECODE (17 bytes)    
Length Header 1 Actual Content Length Header 1 Actual Content
0x00000C 0xCAFE “HELLO, WORLD” 0x00000C 0xCAFE “HELLO, WORLD”

2 bytes length field at offset 1 in the middle of 4 bytes header, strip the first header field and the length field This is a combination of all the examples above. There are the prepended header before the length field and the extra header after the length field. The prepended header affects the lengthFieldOffset and the extra header affects the lengthAdjustment. We also specified a non-zero initialBytesToStrip to strip the length field and the prepended header from the frame. If you don’t want to strip the prepended header, you could specify 0 for initialBytesToSkip.

lengthFieldOffset = 1 (= the length of HDR1)

lengthFieldLength = 2

lengthAdjustment = 1 (= the length of HDR2)

initialBytesToStrip = 3 (= the length of HDR1 + LEN)

BEFORE DECODE (16 bytes)       AFTER DECODE (13 bytes)  
HDR1 Length HDR2 Actual Content HDR2 Actual Content
0xCA 0x000C 0xFE “HELLO, WORLD” 0xFE “HELLO, WORLD”

2 bytes length field at offset 1 in the middle of 4 bytes header, strip the first header field and the length field, the length field represents the length of the whole message Let’s give another twist to the previous example. The only difference from the previous example is that the length field represents the length of the whole message instead of the message body, just like the third example. We have to count the length of HDR1 and Length into lengthAdjustment. Please note that we don’t need to take the length of HDR2 into account because the length field already includes the whole header length.

lengthFieldOffset = 1

lengthFieldLength = 2

lengthAdjustment = -3 (= the length of HDR1 + LEN, negative)

initialBytesToStrip = 3

BEFORE DECODE (16 bytes)       AFTER DECODE (13 bytes)  
HDR1 Length HDR2 Actual Content HDR2 Actual Content
0xCA 0x0010 0xFE “HELLO, WORLD” 0xFE “HELLO, WORLD”

LengthFieldBasedFrameDecoder

长度域+消息体，长度域（lengthField）默认代表的是紧跟其后的消息体的长度值

  • maxFrameLength

    最大长度

  • lengthFieldOffset

    长度域开始位置

  • lengthFieldLength

    长度域的长度（字节数），unsigned 8/16/24/32/64 bit integer，一般代表body的长度或在整个消息的长度

  • lengthAdjustment

    长度值修正，有时候长度域不仅仅是body的长度，而是整个message的长度，实际还包含了lengthfield或header1或header2，因而需要调整。实际lengthFieldLength后面内容的长度=lengthFieldLength + lengthAdjustment

    lengthfield或header1：取对应负值；header2：取对应正值

  • initialBytesToStrip

    剥去前面指定字节数量的内容

  • byteOrder

    长度域字节顺序

  • failFast

    是否快速失败

  • lengthFieldEndOffset

    长度域结束位置，等于lengthFieldOffset + lengthFieldLength

LengthFieldBasedFrameDecoder(1024, 0, 2, 0, 0)

则读取到一条业务消息：16QUERY TIME ORDER

LengthFieldBasedFrameDecoder(1024, 0, 2, 0, 2)

则读取到一条业务消息：QUERY TIME ORDER

LengthFieldPrepender

  • lengthFieldLength

    长度域的长度（字节数），unsigned 8/16/24/32/64 bit integer

  • lengthIncludesLengthFieldLength

    长度值是否包含长度域本身，是则实际长度值为body长度加上长度域的长度

  • lengthAdjustment

    长度值修正，实际长度值为body长度加上长度值修正。

  • byteOrder

    长度域字节顺序

ASCII编码长度解码

可以覆盖LengthFieldBasedFrameDecoder之解析长度的方法，长度域长度类型如果为ascii，参考如下：

1
2
3
4
5
protected long getUnadjustedFrameLength(ByteBuf buf, int offset, int length, ByteOrder order){
	byte[] len = new byte[length];
	buf.getBytes(offset, len);
	return Long.parseLong(new String(len, StandardCharsets.US_ASCII));
}

发送方发送数据：

1
"16QUERY TIME ORDER16QUERY TIME ORDER16QUERY TIME ORDER".getBytes()

备注：长度域ascii编码，一个ascii字符一个字节

ASCII编码长度编码

可以参照LengthFieldPrepender，实现MessageToMessageEncoder<ByteBuf>，重点方法如下：

1
2
3
4
5
6
@Override
    protected void encode(ChannelHandlerContext ctx, ByteBuf msg, List<Object> out) throws Exception {
        int length = msg.readableBytes();
        out.add(ctx.alloc().buffer(lengthFieldLength).writeBytes(String.format("%0"+lengthFieldLength+"d", length).getBytes(StandardCharsets.US_ASCII)));
        out.add(msg.retain());
    }