色哟哟视频在线观看-色哟哟视频在线-色哟哟欧美15最新在线-色哟哟免费在线观看-国产l精品国产亚洲区在线观看-国产l精品国产亚洲区久久

0
  • 聊天消息
  • 系統消息
  • 評論與回復
登錄后你可以
  • 下載海量資料
  • 學習在線課程
  • 觀看技術視頻
  • 寫文章/發帖/加入社區
會員中心
創作中心

完善資料讓更多小伙伴認識你,還能領取20積分哦,立即完善>

3天內不再提示

一文詳解Linux內存檢測技術

Linux內核補給站 ? 來源:Linux內核補給站 ? 作者:Linux內核補給站 ? 2022-05-20 18:00 ? 次閱讀

Linux常見的內存訪問錯誤有:

越界訪問(out of bounds)

訪問已經釋放的內存(use after free)

重復釋放

內存泄露(memory leak)

棧溢出(stack overflow)

不同的工具有不同的側重點,本章主要從slub_debug、kmemleak、kasan三個工具介紹。

kmemleak側重于內存泄露問題發現。

slub_debug和kasan有一定的重復,部分slub_debug問題需要借助slabinfo去發現;kasan更快,所有問題獨立上報,缺點是需要高版本GCC支持(gcc 4.9.2 or gcc 5.0)。

測試環境準備

更新內核版本到Kernel v4.4,然后編譯:

git clone https://github.com/arnoldlu/linux.git -b running_kernel_4.4 export ARCH=arm64 export CROSS_COMPILE=aarch64-linux-gnu- make defconfig make bzImage -j4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

slub_debug

關鍵詞:Red Zone、Padding、Object Layout。

Linux內核中,小塊內存大量使用slab/slub分配器,slub_debug提供了內存檢測小功能。

內存中比較容易出錯的地方有:

訪問已經釋放的內存

越界訪問

重復釋放內存

編譯支持slub_debug內核

首先需要打開General setup -> Enable SLUB debugging support,然后再選擇Kernel hacking -> Memory Debugging -> SLUB debugging on by default。

CONFIG_SLUB=y CONFIG_SLUB_DEBUG=y CONFIG_SLUB_DEBUG_ON=y CONFIG_SLUB_STATS=y

測試環境:slabinfo、slub.ko

通過slub.ko模擬內存異常訪問,有些可以直接顯示,有些需要通過slabinfo -v來查看。

在tools/vm目錄下,執行如下命令,生成可執行文件slabinfo。放入_install目錄,打包到zImage中。

make slabinfo CFLAGS=-static ARCH=arm CROSS_COMPILE=arm-linux-gnueabi-

將編譯好的slabinfo放入sbin。

下面三個測試代碼:https://github.com/arnoldlu/linux/tree/running_kernel_4.4/test_code/slub_debug

在test_code/slub_debug目錄下執行make.sh,將slub.ko/slub2.ko/slub3.ko放入data。

進行測試

啟動QEMU:

qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -smp 2 -m 2048 -kernel arch/arm64/boot/Image --append "rdinit=/linuxrc console=ttyAMA0 loglevel=8 slub_debug=UFPZ" -nographic

F:在free的時候會執行檢查。

Z:表示Red Zone的意思。

P:是Poison的意思。

U:會記錄slab的使用者信息,如果打開,會會顯示分配釋放對象的棧回溯。

在slub_debug打開SLAB_STORE_USER選項后,可以清晰地看到問題點的backtrace。

測試結果

內存越界訪問包括Redzone overwritten和Object padding overwritten。

重復釋放對應Object already free。訪問已釋放內存為Posion overwritten。

Redzone overwritten

執行insmod data/slub.ko,使用slabinfo -v查看結果。

static void create_slub_error(void) { buf = kmalloc(32, GFP_KERNEL); if(buf) { memset(buf, 0x55, 80);-----------------------------------雖然分配32字節,但是對應分配了64字節。所以設置為80字節訪問觸發異常。從buf開始的80個字節仍然被初始化成功。 } }

雖然kmalloc申請了32字節的slab緩沖區,但是內核分配的是kmalloc-64。所以memset 36字節不會報錯,將36改成大于64即可。

一個slub Debug輸出包括四大部分:

=============================================================================

BUG kmalloc-64 (Tainted: G O ): Redzone overwritten-------------------------------------------------------------1. 問題描述:slab名稱-kmalloc-64,什么錯誤-Redzone overwritten。
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: 0xeddb3640-0xeddb3643. First byte 0x55 instead of 0xcc------------------------------------------------1.1 問題起始和結束地址,這里一共4字節。
INFO: Allocated in 0x55555555 age=1766 cpu=0 pid=771---------------------------------------------------------1.2 slab的分配棧回溯
0x55555555
0xbf002014
do_one_initcall+0x90/0x1d8
do_init_module+0x60/0x38c
load_module+0x1bac/0x1e94
SyS_init_module+0x14c/0x15c
ret_fast_syscall+0x0/0x3c
INFO: Freed in do_one_initcall+0x78/0x1d8 age=1766 cpu=0 pid=771-----------------------------------------1.3 slab的釋放棧回溯
do_one_initcall+0x78/0x1d8
do_init_module+0x60/0x38c
load_module+0x1bac/0x1e94
SyS_init_module+0x14c/0x15c
ret_fast_syscall+0x0/0x3c
INFO: Slab 0xefdb5660 objects=16 used=14 fp=0xeddb3700 flags=0x0081-----------------------------------1.4 slab的地址,以及其它信息。
INFO: Object 0xeddb3600 @offset=1536 fp=0x55555555-----------------------------------------------------------1.5 當前Object起始,及相關信息

Bytes b4 eddb35f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ------------2. 問題slab對象內容。2.1 打印問題slab對象內容之前一些字節。
Object eddb3600: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU---------2.2 slab對象內容,全部為0x55。
Object eddb3610: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU
Object eddb3620: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU
Object eddb3630: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU
Redzone eddb3640: 55 55 55 55 UUUU----------------------------------------------------------------------------------2.3 Redzone內容,問題出在這里。
Padding eddb36e8: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ------------2.4 Padding內容,為了對象對齊而補充。
Padding eddb36f8: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
CPU: 2 PID: 773 Comm: slabinfo Tainted: G B O 4.4.0+ #93--------------------------------------------------------3. 檢查問題點的棧打印,這里是由于slabinfo找出來的。
Hardware name: ARM-Versatile Express
[] (unwind_backtrace) from [] (show_stack+0x10/0x14)
[] (show_stack) from [] (dump_stack+0x78/0x88)
[] (dump_stack) from [] (check_bytes_and_report+0xd0/0x10c)
[] (check_bytes_and_report) from [] (check_object+0x164/0x234)
[] (check_object) from [] (validate_slab_slab+0x198/0x1bc)
[] (validate_slab_slab) from [] (validate_store+0xac/0x190)
[] (validate_store) from [] (kernfs_fop_write+0xb8/0x1b4)
[] (kernfs_fop_write) from [] (__vfs_write+0x1c/0xd8)
[] (__vfs_write) from [] (vfs_write+0x90/0x170)
[] (vfs_write) from [] (SyS_write+0x3c/0x90)
[] (SyS_write) from [] (ret_fast_syscall+0x0/0x3c)
FIX kmalloc-64: Restoring 0xeddb3640-0xeddb3643=0xcc----------------------------------------------------------4. 問題點是如何被解決的,此處恢復4個字節為0xcc。

Object padding overwritten

void create_slub_error(void)
{
  int i;

  buf = kmalloc(32, GFP_KERNEL);
  if(buf) {
    buf[-1] = 0x55;------------------------------------------------------------------------向左越界訪問
    kfree(buf);
  }
}
執行insmod data/slub4.ko,結果如下。
這里的越界訪問和之前有點不一樣的是,這里向左越界。覆蓋到了Padding區域。
al: slub error test init
=============================================================================
BUG kmalloc-128 (Tainted: G O ): Object padding overwritten------------------------------------------------------覆蓋到Padding區域
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: 0xffff80007767e9ff-0xffff80007767e9ff. First byte 0x55 instead of 0x5a
INFO: Allocated in call_usermodehelper_setup+0x44/0xb8 age=1 cpu=1 pid=789
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x1ec/0x260
call_usermodehelper_setup+0x44/0xb8
/ # kobject_uevent_env+0x494/0x500
kobject_uevent+0x10/0x18
load_module+0x18cc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2dd9f80 objects=16 used=9 fp=0xffff80007767ea00 flags=0x4081
INFO: Object 0xffff80007767e800 @offset=2048 fp=0xffff80007767ea00

Bytes b4 ffff80007767e7f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Object ffff80007767e800: 00 01 00 00 00 00 00 00 08 e8 67 77 00 80 ff ff ..........gw....
Object ffff80007767e810: 08 e8 67 77 00 80 ff ff f8 83 0c 00 00 80 ff ff ..gw............
Object ffff80007767e820: 00 00 00 00 00 00 00 00 00 6e aa 00 00 80 ff ff .........n......
Object ffff80007767e830: 00 23 67 78 00 80 ff ff 18 23 67 78 00 80 ff ff .#gx.....#gx....
Object ffff80007767e840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff80007767e850: b8 8e 32 00 00 80 ff ff 00 23 67 78 00 80 ff ff ..2......#gx....
Object ffff80007767e860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff80007767e870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Redzone ffff80007767e880: cc cc cc cc cc cc cc cc ........
Padding ffff80007767e9c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff80007767e9d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff80007767e9e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff80007767e9f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 55 ZZZZZZZZZZZZZZZU
CPU: 0 PID: 790 Comm: mdev Tainted: G B O 4.4.0+ #116
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x108
[] show_stack+0x14/0x20
[] dump_stack+0x94/0xd0
[] print_trailer+0x128/0x1b8
[] check_bytes_and_report+0xd8/0x118
[] check_object+0xa0/0x240
[] free_debug_processing+0x128/0x380
[] __slab_free+0x344/0x4a0
[] kfree+0x1ec/0x220
[] umh_complete+0x58/0x68
[] call_usermodehelper_exec_async+0x150/0x170
[] ret_from_fork+0x10/0x40
FIX kmalloc-128: Restoring 0xffff80007767e9ff-0xffff80007767e9ff=0x5a---------------------------------------------------------問題處理是將對應字節恢復為0x5a。

Object already free

void create_slub_error(void)
{
  buf = kmalloc(32, GFP_KERNEL);
  if(buf) {
    memset(buf, 0x55, 32);
    kfree(buf);
    printk("al: Object already freed");
    kfree(buf);
  }
}
內核中free執行流程如下:
kfree
  ->slab_free
    ->__slab_free
      ->kmem_cache_debug
        ->free_debug_processing
         ->on_freelist   
執行insmod data/slub2.ko,結果如下。
al: slub error test init
al: Object already freed
=============================================================================
BUG kmalloc-128 (Tainted: G B O ): Object already free------------------------------------------------------------------在64位系統,32字節的kmalloc變成了kmalloc-128,問題類型是:Object already free,也即重復釋放。
-----------------------------------------------------------------------------

INFO: Allocated in create_slub_error+0x20/0x80 [slub2] age=0 cpu=1 pid=791------------------------------------內存分配點棧回溯
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x1ec/0x260
create_slub_error+0x20/0x80 [slub2]
my_test_init+0x14/0x28 [slub2]
do_one_initcall+0x90/0x1a0
do_init_module+0x60/0x1cc
load_module+0x18dc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Freed in create_slub_error+0x50/0x80 [slub2] age=0 cpu=1 pid=791------------------------------------------內存釋放點棧回溯
free_debug_processing+0x17c/0x380
__slab_free+0x344/0x4a0
kfree+0x1ec/0x220
create_slub_error+0x50/0x80 [slub2]
my_test_init+0x14/0x28 [slub2]
do_one_initcall+0x90/0x1a0
do_init_module+0x60/0x1cc
load_module+0x18dc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2dda800 objects=16 used=7 fp=0xffff8000776a0800 flags=0x4081
INFO: Object 0xffff8000776a0800 @offset=2048 fp=0xffff8000776a0a00

Bytes b4 ffff8000776a07f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Object ffff8000776a0800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk-----------------內存內容打印,供128字節。
Object ffff8000776a0810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff8000776a0870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
Redzone ffff8000776a0880: bb bb bb bb bb bb bb bb ........
Padding ffff8000776a09c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776a09d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776a09e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776a09f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
CPU: 1 PID: 791 Comm: insmod Tainted: G B O 4.4.0+ #116--------------------------------------------------------------此處問題在insmod就發現了,所以檢查出問題的進程就是insmod。
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x108
[] show_stack+0x14/0x20
[] dump_stack+0x94/0xd0
[] print_trailer+0x128/0x1b8
[] free_debug_processing+0x29c/0x380
[] __slab_free+0x344/0x4a0
[] kfree+0x1ec/0x220
[] create_slub_error+0x60/0x80 [slub2]
[] my_test_init+0x14/0x28 [slub2]
[] do_one_initcall+0x90/0x1a0
[] do_init_module+0x60/0x1cc
[] load_module+0x18dc/0x1d78
[] SyS_init_module+0x150/0x178
[] el0_svc_naked+0x24/0x28
FIX kmalloc-128: Object at 0xffff8000776a0800 not freed------------------------------------------------------------------處理的結果是,此處slab 對象是沒有被釋放。

Poison overwritten

訪問已釋放內存的測試代碼如下:

static void create_slub_error(void)
{
  buf = kmalloc(32, GFP_KERNEL);-----------------------此時的buf內容都是0x6B
  if(buf) {
    kfree(buf);
    printk("al: Access after free");
    memset(buf, 0x55, 32);-----------------------------雖然被釋放,但是memset仍然生效了變成了0x55。
  }
}
執行insmod data/slub3.ko ,使用slabinfo -v查看結果。
=============================================================================

BUG kmalloc-128 (Tainted: G B O ): Poison overwritten----------------------------------------------slab名稱為kmalloc-64,問題類型是:Poison overwritten,即訪問已釋放內存。
-----------------------------------------------------------------------------

 

INFO: 0xffff800077692800-0xffff80007769281f. First byte 0x55 instead of 0x6b
INFO: Allocated in create_slub_error+0x28/0xf0 [slub3] age=1089 cpu=1 pid=793----------分配點的棧回溯
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x1ec/0x260
create_slub_error+0x28/0xf0 [slub3]
0xffff7ffffc00e014
do_one_initcall+0x90/0x1a0
do_init_module+0x60/0x1cc
load_module+0x18dc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Freed in create_slub_error+0x80/0xf0 [slub3] age=1089 cpu=1 pid=793--------------釋放點的棧回溯
free_debug_processing+0x17c/0x380
__slab_free+0x344/0x4a0
kfree+0x1ec/0x220
create_slub_error+0x80/0xf0 [slub3]
0xffff7ffffc00e014
do_one_initcall+0x90/0x1a0
do_init_module+0x60/0x1cc
load_module+0x18dc/0x1d78
SyS_init_module+0x150/0x178
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2dda480 objects=16 used=16 fp=0x (null) flags=0x4080
INFO: Object 0xffff800077692800 @offset=2048 fp=0xffff800077692400

 

Bytes b4 ffff8000776927f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Object ffff800077692800: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU--------前32字節仍然被修改成功。
Object ffff800077692810: 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 UUUUUUUUUUUUUUUU
Object ffff800077692820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff800077692870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk.
Redzone ffff800077692880: bb bb bb bb bb bb bb bb ........
Padding ffff8000776929c0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776929d0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776929e0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
Padding ffff8000776929f0: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZZZZZZZZZ
CPU: 0 PID: 795 Comm: slabinfo Tainted: G B O 4.4.0+ #116
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x108
[] show_stack+0x14/0x20
[] dump_stack+0x94/0xd0
[] print_trailer+0x128/0x1b8
[] check_bytes_and_report+0xd8/0x118
[] check_object+0x1cc/0x240
[] alloc_debug_processing+0x108/0x188
[] ___slab_alloc.constprop.30+0x3f8/0x440
[] __slab_alloc.isra.27.constprop.29+0x24/0x38
[] kmem_cache_alloc+0x1ec/0x260
[] seq_open+0x34/0x90
[] kernfs_fop_open+0x194/0x370
[] do_dentry_open+0x214/0x318
[] vfs_open+0x58/0x68
[] path_openat+0x460/0xdf0
[] do_filp_open+0x60/0xe0
[] do_sys_open+0x12c/0x218
[] compat_SyS_open+0x1c/0x28
[] el0_svc_naked+0x24/0x28
FIX kmalloc-128: Restoring 0xffff800077692800-0xffff80007769281f=0x6b

 

FIX kmalloc-128: Marking all objects used
SLUB: kmalloc-128 210 slabs counted but counter=211
slabinfo (795) used greatest stack depth: 12976 bytes left

kmemleak

kmemleak是內核提供的一種檢測內存泄露工具,啟動一個內核線程掃描內存,并打印發現新的未引用對象數量。

支持kmemleak內核選項

要使用kmemlieak,需要打開如下內核選項。

Kernel hacking->Memory Debugging->Kernel memory leak detector:

CONFIG_HAVE_DEBUG_KMEMLEAK=y CONFIG_DEBUG_KMEMLEAK=y CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=400 # CONFIG_DEBUG_KMEMLEAK_TEST is not set CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y---------或者關閉此選項,則不需要在命令行添加kmemleak=on。

構造測試環境

同時還需要在內核啟動命令行中添加kmemleak=on。

qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -smp 2 -m 2048 -kernel arch/arm64/boot/Image --append "rdinit=/linuxrc console=ttyAMA0 loglevel=8 kmemleak=on" -nographic

static char *buf; void create_kmemleak(void) { buf = kmalloc(120, GFP_KERNEL); buf = vmalloc(4096); }

進行測試

進行kmemleak測試之前,需要寫入scan觸發掃描操作。

然后通過讀kmemlean節點讀取相關信息。

打開kmemlean掃描功能:echo scan > sys/kernel/debug/kmemleak

加載問題module:insmod data/kmemleak.ko

等待問題發現:kmemleak: 2 new suspected memory leaks (see /sys/kernel/debug/kmemleak)

查看kmemleak結果:cat /sys/kernel/debug/kmemleak

分析測試結果

每處泄露,都標出泄露地址和大小;相關進程信息;內存內容dump;棧回溯。
kmemleak會提示內存泄露可疑對象的具體棧調用信息、可疑對象的大小、使用哪個函數分配、二進制打印。
unreferenced object 0xede22dc0 (size 128):-------------------------------------第一處可疑泄露128字節
  comm "insmod", pid 765, jiffies 4294941257 (age 104.920s)--------------------相關進程信息
  hex dump (first 32 bytes):---------------------------------------------------二進制打印
    6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
    6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
  backtrace:-------------------------------------------------------------------棧回溯
    [] 0xbf002014
    [] do_one_initcall+0x90/0x1d8
    [] do_init_module+0x60/0x38c
    [] load_module+0x1bac/0x1e94
    [] SyS_init_module+0x14c/0x15c
    [] ret_fast_syscall+0x0/0x3c
    [] 0xffffffff
unreferenced object 0xf12ba000 (size 4096):
  comm "insmod", pid 765, jiffies 4294941257 (age 104.920s)
  hex dump (first 32 bytes):
    d8 21 00 00 02 18 00 00 e4 21 00 00 02 18 00 00  .!.......!......
    46 22 00 00 02 18 00 00 52 22 00 00 02 18 00 00  F"......R"......
  backtrace:
    [] vmalloc+0x2c/0x34
    [] 0xbf002014
    [] do_one_initcall+0x90/0x1d8
    [] do_init_module+0x60/0x38c
    [] load_module+0x1bac/0x1e94
    [] SyS_init_module+0x14c/0x15c
    [] ret_fast_syscall+0x0/0x3c
    [] 0xffffffff

kasan

kasan暫不支持32位ARM,支持ARM64和X86。

kasan是一個動態檢查內存錯誤的工具,可以檢查內存越界訪問、使用已釋放內存、重復釋放以及棧溢出。

使能kasan

使用kasan,必須打開CONFIG_KASAN。

Kernel hacking->Memory debugging->KASan: runtime memory debugger

CONFIG_HAVE_ARCH_KASAN=y CONFIG_KASAN=y # CONFIG_KASAN_OUTLINE is not set CONFIG_KASAN_INLINE=y CONFIG_TEST_KASAN=m

代碼分析

kasan_report ->kasan_report_error ->print_error_description ->print_address_description ->print_shadow_for_address

測試用及分析

kasan提供了一個測試程序test_kacan.c,將其編譯成模塊,加載到內核。可以模擬很多內存錯誤場景。

kasan可以檢測到越界訪問、訪問已釋放內存、重復釋放等類型錯誤,其中重復釋放借助于slub_debug。

insmod data/kasan.ko

越界訪問包括slab越界、棧越界、全局變量越界;訪問已釋放內存use-after-free;重復釋放可以被slub_debug識別。

slab-out-of-bounds

static noinline void __init kmalloc_oob_right(void)
{
    char *ptr;
    size_t size = 123;

    pr_info("out-of-bounds to right\n");
    ptr = kmalloc(size, GFP_KERNEL);
    if (!ptr) {
        pr_err("Allocation failed\n");
        return;
    }

    ptr[size] = 'x';
    kfree(ptr);
}
此種錯誤類型是對slab的越界訪問,包括左側、右側、擴大、縮小后越界訪問。除了數組賦值,還包括memset、指針訪問等等。
al: kasan error test init
kasan test: kmalloc_oob_right out-of-bounds to right
==================================================================
BUG: KASAN: slab-out-of-bounds in kmalloc_oob_right+0xa4/0xe0 [kasan] at addr ffff800066539c7b----------------錯誤類型是slab-out-of-bounds,在kmalloc_oob_right中產生。
Write of size 1 by task insmod/788
=============================================================================
BUG kmalloc-128 (Tainted: G O ): kasan: bad access detected-------------------------------------------------------------------slab非法非法訪問
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: Allocated in kmalloc_oob_right+0x54/0xe0 [kasan] age=0 cpu=1 pid=788--------------------------------------------問題點kmalloc_oob_right的棧回溯
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x220/0x280
kmalloc_oob_right+0x54/0xe0 [kasan]
kmalloc_tests_init+0x18/0x70 [kasan]
do_one_initcall+0x11c/0x310
do_init_module+0x1cc/0x588
load_module+0x48cc/0x5dc0
SyS_init_module+0x1a8/0x1e0
el0_svc_naked+0x24/0x28
INFO: Freed in do_one_initcall+0x10c/0x310 age=0 cpu=1 pid=788
free_debug_processing+0x17c/0x368
__slab_free+0x344/0x4a0
kfree+0x21c/0x250
do_one_initcall+0x10c/0x310
do_init_module+0x1cc/0x588
load_module+0x48cc/0x5dc0
SyS_init_module+0x1a8/0x1e0
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2994e00 objects=16 used=2 fp=0xffff800066539e00 flags=0x4080
INFO: Object 0xffff800066539c00 @offset=7168 fp=0xffff800066538200

Bytes b4 ffff800066539bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................------------------------------內存dump
Object ffff800066539c00: 00 82 53 66 00 80 ff ff 74 65 73 74 73 5f 69 6e ..Sf....tests_in
Object ffff800066539c10: 69 74 20 5b 6b 61 73 61 6e 5d 00 00 00 00 00 00 it [kasan]......
Object ffff800066539c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
CPU: 1 PID: 788 Comm: insmod Tainted: G B O 4.4.0+ #108------------------------------------------------------------------打印此log消息的棧回溯
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x270
[] show_stack+0x14/0x20
[] dump_stack+0x100/0x188
[] print_trailer+0xf8/0x160
[] object_err+0x3c/0x50
[] kasan_report_error+0x240/0x558
[] __asan_report_store1_noabort+0x48/0x50
[] kmalloc_oob_right+0xa4/0xe0 [kasan]
[] kmalloc_tests_init+0x18/0x70 [kasan]
[] do_one_initcall+0x11c/0x310
[] do_init_module+0x1cc/0x588
[] load_module+0x48cc/0x5dc0
[] SyS_init_module+0x1a8/0x1e0
[] el0_svc_naked+0x24/0x28
Memory state around the buggy address:
ffff800066539b00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff800066539b80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>ffff800066539c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03
^
ffff800066539c80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff800066539d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================

user-after-free

user-after-free是釋放后使用的意思。

static noinline void __init kmalloc_uaf(void)
{
    char *ptr;
    size_t size = 10;

    pr_info("use-after-free\n");
    ptr = kmalloc(size, GFP_KERNEL);
    if (!ptr) {
        pr_err("Allocation failed\n");
        return;
    }

    kfree(ptr);
    *(ptr + 8) = 'x';
}

測試結果如下:

kasan test: kmalloc_uaf use-after-free
==================================================================
BUG: KASAN: use-after-free in kmalloc_uaf+0xac/0xe0 [kasan] at addr ffff800066539e08
Write of size 1 by task insmod/788
=============================================================================
BUG kmalloc-128 (Tainted: G B O ): kasan: bad access detected
-----------------------------------------------------------------------------

INFO: Allocated in kmalloc_uaf+0x54/0xe0 [kasan] age=0 cpu=1 pid=788
alloc_debug_processing+0x17c/0x188
___slab_alloc.constprop.30+0x3f8/0x440
__slab_alloc.isra.27.constprop.29+0x24/0x38
kmem_cache_alloc+0x220/0x280
kmalloc_uaf+0x54/0xe0 [kasan]
kmalloc_tests_init+0x48/0x70 [kasan]
do_one_initcall+0x11c/0x310
do_init_module+0x1cc/0x588
load_module+0x48cc/0x5dc0
SyS_init_module+0x1a8/0x1e0
el0_svc_naked+0x24/0x28
INFO: Freed in kmalloc_uaf+0x84/0xe0 [kasan] age=0 cpu=1 pid=788
free_debug_processing+0x17c/0x368
__slab_free+0x344/0x4a0
kfree+0x21c/0x250
kmalloc_uaf+0x84/0xe0 [kasan]
kmalloc_tests_init+0x48/0x70 [kasan]
do_one_initcall+0x11c/0x310
do_init_module+0x1cc/0x588
load_module+0x48cc/0x5dc0
SyS_init_module+0x1a8/0x1e0
el0_svc_naked+0x24/0x28
INFO: Slab 0xffff7bffc2994e00 objects=16 used=1 fp=0xffff800066539e00 flags=0x4080
INFO: Object 0xffff800066539e00 @offset=7680 fp=0xffff800066539800

Bytes b4 ffff800066539df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539e00: 00 98 53 66 00 80 ff ff 00 00 00 00 00 00 00 00 ..Sf............
Object ffff800066539e10: 00 9e 53 66 00 80 ff ff d0 51 12 00 00 80 ff ff ..Sf.....Q......
Object ffff800066539e20: 00 00 00 00 00 00 00 00 e0 14 6d 01 00 80 ff ff ..........m.....
Object ffff800066539e30: 00 69 a3 66 00 80 ff ff 18 69 a3 66 00 80 ff ff .i.f.....i.f....
Object ffff800066539e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539e50: 30 da 73 00 00 80 ff ff 00 69 a3 66 00 80 ff ff 0.s......i.f....
Object ffff800066539e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object ffff800066539e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Padding ffff800066539ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
CPU: 1 PID: 788 Comm: insmod Tainted: G B O 4.4.0+ #108
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x270
[] show_stack+0x14/0x20
[] dump_stack+0x100/0x188
[] print_trailer+0xf8/0x160
[] object_err+0x3c/0x50
[] kasan_report_error+0x240/0x558
[] __asan_report_store1_noabort+0x48/0x50
[] kmalloc_uaf+0xac/0xe0 [kasan]
[] kmalloc_tests_init+0x48/0x70 [kasan]
[] do_one_initcall+0x11c/0x310
[] do_init_module+0x1cc/0x588
[] load_module+0x48cc/0x5dc0
[] SyS_init_module+0x1a8/0x1e0
[] el0_svc_naked+0x24/0x28
Memory state around the buggy address:
ffff800066539d00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff800066539d80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>ffff800066539e00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff800066539e80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff800066539f00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
==================================================================

stack-out-of-bounds

棧越界訪問是函數中數組越界,在實際工程中經常出現,問題難以發現。

static noinline void __init kasan_stack_oob(void)
{
    char stack_array[10];
    volatile int i = 0;
    char *p = &stack_array[ARRAY_SIZE(stack_array) + i];

    pr_info("out-of-bounds on stack\n");
    *(volatile char *)p;
}
 




kasan test: kasan_stack_oob out-of-bounds on stack
==================================================================
BUG: KASAN: stack-out-of-bounds in kasan_stack_oob+0xa8/0xf0 [kasan] at addr ffff800066acb95a
Read of size 1 by task insmod/788
page:ffff7bffc29ab2c0 count:0 mapcount:0 mapping: (null) index:0x0
flags: 0x0()
page dumped because: kasan: bad access detected
CPU: 1 PID: 788 Comm: insmod Tainted: G B O 4.4.0+ #108
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x270
[] show_stack+0x14/0x20
[] dump_stack+0x100/0x188
[] kasan_report_error+0x530/0x558
[] __asan_report_load1_noabort+0x48/0x50
[] kasan_stack_oob+0xa8/0xf0 [kasan]
[] kmalloc_tests_init+0x58/0x70 [kasan]
[] do_one_initcall+0x11c/0x310
[] do_init_module+0x1cc/0x588
[] load_module+0x48cc/0x5dc0
[] SyS_init_module+0x1a8/0x1e0
[] el0_svc_naked+0x24/0x28
Memory state around the buggy address:
ffff800066acb800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff800066acb880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
>ffff800066acb900: f1 f1 04 f4 f4 f4 f2 f2 f2 f2 00 02 f4 f4 f3 f3
^
ffff800066acb980: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
ffff800066acba00: f1 f1 00 00 00 00 00 00 00 00 f3 f3 f3 f3 00 00
==================================================================

global-out-of-bounds

static char global_array[10];

static noinline void __init kasan_global_oob(void)
{
    volatile int i = 3;
    char *p = &global_array[ARRAY_SIZE(global_array) + i];

    pr_info("out-of-bounds global variable\n");
    *(volatile char *)p;
}

測試結果如下:

kasan test: kasan_global_oob out-of-bounds global variable
==================================================================
BUG: KASAN: global-out-of-bounds in kasan_global_oob+0x9c/0xe8 [kasan] at addr ffff7ffffc001c8d
Read of size 1 by task insmod/788
Address belongs to variable global_array+0xd/0xffffffffffffe3f8 [kasan]
CPU: 1 PID: 788 Comm: insmod Tainted: G B O 4.4.0+ #108
Hardware name: linux,dummy-virt (DT)
Call trace:
[] dump_backtrace+0x0/0x270
[] show_stack+0x14/0x20
[] dump_stack+0x100/0x188
[] kasan_report_error+0x530/0x558
[] __asan_report_load1_noabort+0x48/0x50
[] kasan_global_oob+0x9c/0xe8 [kasan]
[] kmalloc_tests_init+0x5c/0x70 [kasan]
[] do_one_initcall+0x11c/0x310
[] do_init_module+0x1cc/0x588
[] load_module+0x48cc/0x5dc0
[] SyS_init_module+0x1a8/0x1e0
[] el0_svc_naked+0x24/0x28
Memory state around the buggy address:
ffff7ffffc001b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff7ffffc001c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>ffff7ffffc001c80: 00 02 fa fa fa fa fa fa 00 00 00 00 00 00 00 00
^
ffff7ffffc001d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ffff7ffffc001d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
==================================================================

小結

kmemleak檢查內存泄露的獨門絕技,讓其有一定市場空間。但功能比較單一,專注于內存泄露問題。

對于非ARM64/x86平臺,只能使用slub_debug進行內存問題分析;kasan更高效,但也需要更高的內核和GCC版本支持。

審核編輯:湯梓紅

聲明:本文內容及配圖由入駐作者撰寫或者入駐合作網站授權轉載。文章觀點僅代表作者本人,不代表電子發燒友網立場。文章及其配圖僅供工程師學習之用,如有內容侵權或者其他違規問題,請聯系本站處理。 舉報投訴
  • Linux
    +關注

    關注

    87

    文章

    11320

    瀏覽量

    209845
  • 內存檢測
    +關注

    關注

    0

    文章

    2

    瀏覽量

    5979
  • 內存管理
    +關注

    關注

    0

    文章

    168

    瀏覽量

    14160
收藏 人收藏

    評論

    相關推薦

    Linux內存管理是什么,Linux內存管理詳解

    Linux內存管理 Linux內存管理是個非常復雜的過程,主要分成兩個大的部分:內核的內存
    的頭像 發表于 05-11 17:54 ?6097次閱讀
    <b class='flag-5'>Linux</b>的<b class='flag-5'>內存</b>管理是什么,<b class='flag-5'>Linux</b>的<b class='flag-5'>內存</b>管理<b class='flag-5'>詳解</b>

    詳解Linux虛擬內存技術

      以存儲單元為單位來管理顯然不現實,因此Linux把虛存空間分成若干個大小相等的存儲分區,Linux把這樣的分區叫做頁。為了換入、換出的方便,物理內存也就按頁的大小分成若干個塊。由于物理內存
    發表于 07-17 17:29 ?594次閱讀
    <b class='flag-5'>一</b><b class='flag-5'>文</b><b class='flag-5'>詳解</b><b class='flag-5'>Linux</b>虛擬<b class='flag-5'>內存</b><b class='flag-5'>技術</b>

    Linux內核的物理內存組織結構詳解

    Linux內存管理子系統使用 節點(node)、區域(zone)和頁(page) 三級結構描述物理內存
    發表于 08-21 15:35 ?587次閱讀
    <b class='flag-5'>Linux</b>內核的物理<b class='flag-5'>內存</b>組織結構<b class='flag-5'>詳解</b>

    Linux內核的內存管理詳解

    內存管理的主要工作就是對物理內存進行組織,然后對物理內存的分配和回收。但是Linux引入了虛擬地址的概念。
    發表于 08-31 14:46 ?802次閱讀
    <b class='flag-5'>Linux</b>內核的<b class='flag-5'>內存</b>管理<b class='flag-5'>詳解</b>

    linux核心技術詳解

    很好的Linux技術詳解,值得讀。
    發表于 11-09 17:35 ?5次下載

    Linux設備驅動開發詳解》第11章、內存與IO訪問

    Linux設備驅動開發詳解》第11章、內存與IO訪問
    發表于 10-27 11:27 ?6次下載
    《<b class='flag-5'>Linux</b>設備驅動開發<b class='flag-5'>詳解</b>》第11章、<b class='flag-5'>內存</b>與IO訪問

    解析Linux內存系統

    Linux 內存是后臺開發人員,需要深入了解的計算機資源。合理的使用內存,有助于提升機器的性能和穩定性。本文主要介紹Linux 內存組織結構
    的頭像 發表于 09-01 10:46 ?2453次閱讀
    <b class='flag-5'>一</b><b class='flag-5'>文</b>解析<b class='flag-5'>Linux</b><b class='flag-5'>內存</b>系統

    詳解linux的分頁模型

    也就是我們實際中編碼時遇到的內存地址并不是對應于實際內存上的地址,我們編碼中使用的地址是個邏輯地址,會通過分段和分頁這兩個機制把它轉為物理地址。而由于linux使用的分段機制有限,可
    的頭像 發表于 05-18 08:59 ?2140次閱讀
    <b class='flag-5'>一</b><b class='flag-5'>文</b><b class='flag-5'>詳解</b><b class='flag-5'>linux</b>的分頁模型

    詳解精密封裝技術

    詳解精密封裝技術
    的頭像 發表于 12-30 15:41 ?1678次閱讀

    詳解分立元件門電路

    詳解分立元件門電路
    的頭像 發表于 03-27 17:44 ?3286次閱讀
    <b class='flag-5'>一</b><b class='flag-5'>文</b><b class='flag-5'>詳解</b>分立元件門電路

    詳解pcb和smt的區別

    詳解pcb和smt的區別
    的頭像 發表于 10-08 09:31 ?3407次閱讀

    詳解pcb地孔的作用

    詳解pcb地孔的作用
    的頭像 發表于 10-30 16:02 ?1701次閱讀

    詳解pcb不良分析

    詳解pcb不良分析
    的頭像 發表于 11-29 17:12 ?1200次閱讀

    詳解pcb的msl等級

    詳解pcb的msl等級
    的頭像 發表于 12-13 16:52 ?9889次閱讀

    詳解pcb微帶線設計

    詳解pcb微帶線設計
    的頭像 發表于 12-14 10:38 ?3427次閱讀
    主站蜘蛛池模板: 亚洲一级特黄| 高清 仑乱 一级 a| 国产睡熟迷奷系列网站| 午理论理影片被窝| 黑丝女仆恋上我| 中国xxxxx| 日本熟妇乱妇熟色A片蜜桃| 国产SUV精品一区二区69| 亚洲精品成人AV在线观看爽翻| 九九99亚洲精品久久久久| 97精品国产自产在线观看永久| 日本高清不卡一区久久精品| 国拍在线精品视频免费观看| 91麻豆久久| 污污内射久久一区二区欧美日韩| 精品一卡2卡三卡4卡乱码精品视频| 97精品在线| 性色AV一区二区三区V视界影院| 久久电影午夜| 俄罗斯大白屁股| 怡春院院日本一区二区久久| 青娱乐在线一区| 黄色三级图片| 把腿张开老子CAO烂你动态图| 亚洲久热无码中文字幕| 欧美一区二区三区男同| 交换邻居波多野结衣中文字幕| MMM日本兽交| 伊人久久影院| 天天躁夜夜踩很很踩2022 | 午夜婷婷精品午夜无码A片影院| 久久精品国产亚洲AV蜜臀| 床伴在线观看免费高清完整泰剧第四集 | 亚洲国产精品久久无套麻豆| 男女啪啪久久精品亚洲A| 国内精品伊人久久久久| 宝贝乖女好紧好深好爽老师| 再深点灬舒服灬太大了在线视频| 睡觉被偷偷进入magnet| 女警被黑人20厘米强交| 久草在线福利资站免费视频 |