http://iet.metastore.ingenta.com
1887

Survey on memory management techniques in heterogeneous computing systems

Survey on memory management techniques in heterogeneous computing systems

For access to this article, please select a purchase option:

Buy article PDF
$19.95
(plus tax if applicable)
Buy Knowledge Pack
10 articles for $120.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Name:*
Email:*
Your details
Name:*
Email:*
Department:*
Why are you recommending this title?
Select reason:
 
 
 
 
 
IET Computers & Digital Techniques — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

A major issue faced by data scientists today is how to scale up their processing infrastructure to meet the challenge of big data and high-performance computing (HPC) workloads. With today's HPC domain, it is required to connect multiple graphics processing units (GPUs) to accomplish large-scale parallel computing along with CPUs. Data movement between the processor and on-chip or off-chip memory creates a major bottleneck in overall system performance. The CPU/GPU processes all the data on a computer's memory and hence the speed of the data movement to/from memory and the size of the memory affect computer speed. During memory access by any processing element, the memory management unit (MMU) controls the data flow of the computer's main memory and impacts the system performance and power. Change in dynamic random access memory (DRAM) architecture, integration of memory-centric hardware accelerator in the heterogeneous system and Processing-in-Memory (PIM) are the techniques adopted from all the available shared resource management techniques to maximise the system throughput. This survey study presents an analysis of various DRAM designs and their performances. The authors also focus on the architecture, functionality, and performance of different hardware accelerators and PIM systems to reduce memory access time. Some insights and potential directions toward enhancements to existing techniques are also discussed. The requirement of fast, reconfigurable, self-adaptive memory management schemes in the high-speed processing scenario motivates us to track the trend. An effective MMU handles memory protection, cache control and bus arbitration associated with the processors.

References

    1. 1)
      • 1. Memory access vs cpu speed’. Available at: http://www.oempcworld.com, accessed September 2019.
    2. 2)
      • 2. Mahapatra, N.R., Venkatrao, B.: ‘The processor-memory bottleneck: problems and solutions’, Crossroads, 1999, 5, (3), pp. 18.
    3. 3)
      • 3. Li, C., Song, S.L., Dai, H., et al: ‘Locality-driven dynamic GPU cache bypassing’. Proc. Int. Conf. Supercomputing, Frankfurt, Germany, 2015, pp. 6777.
    4. 4)
      • 4. Xie, X., Liang, Y., Wang, Y., et al: ‘Coordinated static and dynamic cache bypassing for gpus’. Int. Symp. High Performance Computer Architecture (HPCA), Burlingame, CA, USA, 2015, pp. 7688.
    5. 5)
      • 5. Chen, X., Chang, L.W., Rodrigues, C.I., et al: ‘Adaptive cache management for energy-efficient GPU computing’. Proc. Int. Symp. Microarchitecture, Cambridge, UK, 2014, pp. 343355.
    6. 6)
      • 6. Li, A., van den Braak, G.J., Kumar, A., et al: ‘Adaptive and transparent cache bypassing for GPUs’. Int. Conf. High Performance Computing, Networking, Storage and Analysis, Austin, TX, USA, 2015, pp. 112.
    7. 7)
      • 7. Li, D., Rhu, M., Johnson, D.R., et al: ‘Priority-based cache allocation in throughput processors’. Int. Symp. High Performance Computer Architecture (HPCA), Burlingame, CA, USA, 2015, pp. 89100.
    8. 8)
      • 8. Qureshi, M.K., Jaleel, A., Patt, Y.N., et al: ‘Adaptive insertion policies for high performance caching’, Comput. Archit. News, 2007, 35, (2), pp. 381391.
    9. 9)
      • 9. Rixner, S., Dally, W.J., Kapasi, U.J., et al: ‘Memory access scheduling’, Comput. Archit. News, 2000, 28, (2), pp. 128138.
    10. 10)
      • 10. Ebrahimi, E., Miftakhutdinov, R., Fallin, C., et al: ‘Parallel application memory scheduling’. Proc. Int. Symp. Microarchitecture, Porto Alegre, Brazil, 2011, pp. 362373.
    11. 11)
      • 11. Bakhoda, A., Yuan, G.L., Fung, W.W., et al: ‘Analyzing CUDA workloads using a detailed GPU simulator’. Int. Symp. Performance Analysis of Systems and Software (ISPASS), Boston, MA, USA, 2009, pp. 163174.
    12. 12)
      • 12. Ahn, J.H., Leverich, J., Schreiber, R., et al: ‘Multicore DIMM: an energy efficient memory module with independently controlled DRAMs’, Comput. Archit. Lett., 2009, 8, (1), pp. 58.
    13. 13)
      • 13. Ahn, J.H., Jouppi, N.P., Kozyrakis, C., et al: ‘Improving system energy efficiency with memory rank subsetting’, Trans. Archit. Code Optim., 2012, 9, (1), pp. 4.14.28.
    14. 14)
      • 14. Chandrasekar, K., Goossens, S., Weis, C., et al: ‘Exploiting expendable process-margins in DRAMs for run-time performance optimization’. Proc. Conf. European Design, Automation & Test, Dresden, Germany, 2014, pp. 173178.
    15. 15)
      • 15. Ahn, J., Hong, S., Yoo, S., et al: ‘A scalable processing-in-memory accelerator for parallel graph processing’. Int. Symp. Computer Architecture (ISCA), Portland, OR, USA, 2015, pp. 105117.
    16. 16)
      • 16. Chang, K.K., Nair, P.J., Lee, D., et al: ‘Lowcost inter-linked subarrays (LISA): enabling fast inter-subarray data movement in DRAM’. Int. Symp. High Performance Computer Architecture (HPCA), Barcelona, Spain, 2016, pp. 568580.
    17. 17)
      • 17. Son, Y.H., Seongil, O., Ro, Y., et al: ‘Reducing memory access latency with asymmetric DRAM bank organizations’, Comput. Archit. News, 2013, 41, pp. 380391.
    18. 18)
      • 18. Chatterjee, N., Shevgoor, M., Balasubramonian, R., et al: ‘Leveraging heterogeneity in DRAM main memories to accelerate critical word access’. Int. Symp. Microarchitecture (MICRO), Vancouver, BC, Canada, 2012, pp. 1324.
    19. 19)
      • 19. Seongil, O., Son, Y.H., Kim, N.S., et al: ‘Row-buffer decoupling: a case for low-latency dram microarchitecture’. Int. Symp. Computer Architecture (ISCA), Minneapolis, MN, USA, 2014, pp. 337348.
    20. 20)
      • 20. Karam, R., Paul, S., Puri, R., et al: ‘Memory-centric reconfigurable accelerator for classification and machine learning applications’, J. Emerg. Technol. Comput. Syst., 2017, 13, (3), pp. 34.134.2.
    21. 21)
      • 21. Peemen, M., Setio, A.A., Mesman, B., et al: ‘Memory-centric accelerator design for convolutional neural networks’. Int. Conf. Computer Design (ICCD), Asheville, NC, USA, 2013, pp. 1319.
    22. 22)
      • 22. Che, S., Li, J., Sheaffer, J.W., et al: ‘Accelerating compute-intensive applications with gpus and fpgas’. Symp. Application Specific Processors (SASP), Anaheim, CA USA, 2008, pp. 101107.
    23. 23)
      • 23. Chen, L., Huo, X., Agrawal, G.: ‘Accelerating MapReduce on a coupled CPU-GPU architecture’. Proc. Int. Conf. High Performance Computing, Networking, Storage and Analysis, Los Alamitos, CA, USA, 2012, pp. 2535.
    24. 24)
      • 24. Akin, B., Franchetti, F., Hoe, J.C.: ‘Understanding the design space of dramoptimized hardware fft accelerators’. Int. Conf. Application-specific Systems, Architectures and Processors (ASAP), Zurich, Switzerland, 2014, pp. 248255.
    25. 25)
      • 25. Duarte, F., Wong, S.: ‘Cache-based memory copy hardware accelerator for multicore systems’, IEEE Trans. Comput., 2010, 59, (11), pp. 14941507.
    26. 26)
      • 26. Choi, J., Nam, K., Canis, A., et al: ‘Impact of cache architecture and interface on performance and area of FPGA-based processor/parallel-accelerator systems’. Int. Symp. Field-Programmable Custom Computing Machines (FCCM), Toronto, Canada, 2012, pp. 1724.
    27. 27)
      • 27. Chen, Y.H., Emer, J., Sze, V.: ‘Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks’, Comput. Archit. News, 2016, 44, (3), pp. 367379.
    28. 28)
      • 28. Oliveira, J., Geraldo, F.: ‘A generic processing in memory cycle accurate simulator under hybrid memory cube architecture’. Master thesis, Federal University of Rio Grande do Sul, 2017.
    29. 29)
      • 29. Gokhale, M., Holmes, B., Iobst, K.: ‘Processing in memory: the terasys massively parallel pim array’, Computer. (Long Beach Calif.), 1995, 28, (4), pp. 2331.
    30. 30)
      • 30. Farmahini-Farahani, A., Ahn, J.H., Morrow, K., et al: ‘DRAMA: an architecture for accelerated processing near memory’, Comput. Archit. Lett., 2015, 14, (1), pp. 2629.
    31. 31)
      • 31. Hsieh, K., Khan, S., Vijaykumar, N., et al: ‘Accelerating pointer chasing in 3D-stacked memory: challenges, mechanisms, evaluation’. Int. Conf. Computer Design (ICCD), Phoenix, AZ, USA, 2016, pp. 2532.
    32. 32)
      • 32. Azarkhish, E., Rossi, D., Loi, I., et al: ‘Design and evaluation of a processing-in-memory architecture for the smart memory cube’. Int. Conf. Architecture of Computing Systems, Nuremberg, Germany, 2016, pp. 1931.
    33. 33)
      • 33. Oliveira, G.F., Santos, P.C., Alves, M.A., et al: ‘NIM: an HMC-based machine for neuron computation’. Int. Symp. Applied Reconfigurable Computing, Delft, The Netherlands, 2017, pp. 2835.
    34. 34)
      • 34. Ahn, J., Yoo, S., Mutlu, O., et al: ‘PIM-enabled instructions: a low-overhead, locality-aware processing-in-memory architecture’. Int. Symp. Computer Architecture (ISCA), Portland, MA, USA, 2015, pp. 336348.
    35. 35)
      • 35. Pattnaik, A., Tang, X., Jog, A., et al: ‘Scheduling techniques for GPU architectures with processing-in-memory capabilities’. Int. Conf. Parallel Architecture and Compilation Techniques (PACT), Haifa, Israel, 2016, pp. 3144.
    36. 36)
      • 36. Chi, P., Li, S., Xu, C., et al: ‘PRIME: A novel processing-in-memory architecture for neural network computation in reRAM-based main memory’. Proc. Int. Symp. Computer Architecture, Seoul, Republic of Korea, 2016, pp. 2739.
    37. 37)
      • 37. Zheng, Z., Wen, J., Liu, S.: ‘Introduction to the special section on heterogeneous computing era’, Comput. Electr. Eng., 2017, 60, pp. 4547.
    38. 38)
      • 38. Carretero, J., García-Carballeira, F.: ‘Introduction to special issue on heterogeneous architectures and high-performance computing’, Comput. Electr. Eng., 2013, 8, (39), pp. 25512552.
    39. 39)
      • 39. AMD: ‘Compute cores’. Whitepaper, 2014.
    40. 40)
      • 40. Ausavarungnirun, R., Chang, K.K.W., Subramanian, L., et al: ‘Staged memory scheduling: achieving high performance and scalability in heterogeneous systems’, Comput. Archit. News, 2012, 40, (3), pp. 416427.
    41. 41)
      • 41. NVIDIA: ‘GPU-based deep learning inference: a performance and power analysis’. Whitepaper, 2018.
    42. 42)
      • 42. Davis, B.T.: ‘Modern DRAM architectures’. PhD thesis, University of Michigan, 2001.
    43. 43)
      • 43. Krishnapillai, Y., Wu, Z.P., Pellizzoni, R.: ‘A rank-switching, open-row DRAM controller for time-predictable systems’. Euromicro Conf. Real-Time Systems (ECRTS), Madrid, Spain, 2014, pp. 2738.
    44. 44)
      • 44. Dynamic random-access memory’. Available at: https://en.wikipedia.org/wiki/Dynamicrandomaccessmemory, accessed October 2017.
    45. 45)
      • 45. Reineke, J., Liu, I., Patel, H.D., et al: ‘PRET DRAM controller: bank privatization for predictability and temporal isolation’. Proc. Int. Conf. Hardware/Software Codesign and System Synthesis (CODES+ISSS), Taipei, Taiwan, October 2011, pp. 99108.
    46. 46)
      • 46. Hansson, A., Agarwal, N., Kolli, A., et al: ‘Simulating dram controllers for future system architecture exploration’. Int. Symp. Performance Analysis of Systems and Software (ISPASS), Monterey, CA, USA, 2014, pp. 201210.
    47. 47)
      • 47. Dashti, M., Fedorova, A.: ‘Analyzing memory management methods on integrated CPU-GPU systems’, Memory, 2017, 52, (9), pp. 5969.
    48. 48)
      • 48. Manish, A.: ‘The architecture and evolution of CPU-GPU systems for general purpose computing’. Master thesis, University of California, 2012.
    49. 49)
      • 49. NVIDIA: ‘The benefits of multiple CPU cores in mobile devices’. Whitepaper, 2010.
    50. 50)
      • 50. Technica: ‘GPU accelerated computing’. Whitepaper, 2015.
    51. 51)
      • 51. NVIDIA: ‘NVIDIAs next generation cuda compute architecture: Kepler GK110’. Whitepaper, 2012.
    52. 52)
      • 52. Martindale, J.: ‘AMD Navi graphic cards could offer RTX 2070-like performance for $ 250’. Digitaltrends Whitepaper, 2018.
    53. 53)
      • 53. Han, F., Zhu, T., Meyer, R.: ‘Basic performance analysis of NVIDIA GPU accelerator cards for deep learning applications’. AMAX Whitepaper, 2019.
    54. 54)
      • 54. Varghese, B.: ‘The hardware accelerator debate: a financial risk case study using many-core computing’, Comput. Electr. Eng., 2015, 46, pp. 157175.
    55. 55)
      • 55. Hybrid memory cube specification 1.0’. Tech. Report, Hybrid Memory Cube Consortium, 2013.
    56. 56)
      • 56. Hybrid memory cube specification 2.1’. Tech. Report, Hybrid Memory Cube Consortium, 2014.
    57. 57)
      • 57. Ahn, J., Hong, S., Yoo, S., et al: ‘A scalable processing-in-memory accelerator for parallel graph processing’, Comput. Archit. News, 2016, 43, (3), pp. 105117.
    58. 58)
      • 58. Jin, H.Q., Frumkin, M., Yan, J.: ‘The openmp implementation of NAS parallel benchmarks and its performance’. NASA Ames Research Center, NAS-99-011, 1999.
    59. 59)
      • 59. McGhan, H.: ‘SPEC cpu2006 benchmark suite’. Microprocessor Report, 2006.
    60. 60)
      • 60. SPEC’. Available at: http://www.spec.org/osg/jbb2000/, accessed September 2019.
    61. 61)
      • 61. Beamer, S., Sun, C., Kwon, Y.J., et al: ‘Re-architecting DRAM memory systems with monolithically integrated silicon photonics’, Comput. Archit. News, 2010, 38, pp. 129140.
    62. 62)
      • 62. Chang, K.K.W., Lee, D., Chishti, Z., et al: ‘Improving DRAM performance by parallelizing refreshes with accesses’. Int. Symp. High Performance Computer Architecture (HPCA), Orlando, FL, USA, 2014, pp. 356367.
    63. 63)
      • 63. SPEC CPU2006 benchmarksSPEC (Standard Performance Evaluation Corporation, USA, 2006).
    64. 64)
      • 64. STREAM benchmark’. Available at: http://www.streambench.org/, accessed September 2019.
    65. 65)
      • 65. Tpc benchmarks’. Transaction Performance Processing Council, Available at: http://www.tpc.org/, accessed September 2019.
    66. 66)
      • 66. Dynograph’. Available at: https://github.com/sirpoovey/DynoGraph, accessed September 2019.
    67. 67)
      • 67. Woo, S.C., Ohara, M., Torrie, E., et al: ‘The SPLASH-2 programs: characterization and methodological considerations’. Proc. Int. Symp. Computer Architecture, Santa Margherita Ligure, Italy, 1995, pp. 2436.
    68. 68)
      • 68. Bienia, C., Kumar, S., Singh, J.P., et al: ‘The PARSEC benchmark suite: characterization and architectural implications’. Proc. Int. Conf. Parallel Architectures and Compilation Techniques, Toronto, Canada, 2008, pp. 7281.
    69. 69)
      • 69. All you need to know about mobile LPDRM’. Micron Technology, Available at: https://www.micron.com/products/dram/lpdram, accessed May 2019.
    70. 70)
      • 70. RLDRAM3 press release’. Micron Technology, Available at: http://www.icsi.com.tw/pdf/RLDRAM3-Press-Release.pdf, accessed May 2019.
    71. 71)
      • 71. Bhati, I., Chishti, Z., Lu, S.L., et al: ‘Flexible auto-refresh: enabling scalable and energy-efficient DRAM refresh reductions’, Comput. Archit. News, 2015, 43, (3), pp. 235246.
    72. 72)
      • 72. Nguyen, K., Lyu, K., Meng, X., et al: ‘Nonblocking DRAM refresh’, IEEE Micro, 2019, 39, (3), pp. 103109.
    73. 73)
      • 73. Douglass, A.J., Khatri, S.P.: ‘Fast, ring-based design of 3-D stacked DRAM’, Trans. Very Large Scale Integr. Syst., 2019, 27, (3), pp. 111.
    74. 74)
      • 74. Power, J., Basu, A., Gu, J., et al: ‘Heterogeneous system coherence for integrated CPU-GPU systems’. Proc. Int. Symp. Microarchitecture, Davis, CA, USA, 2013, pp. 457467.
    75. 75)
      • 75. Hechtman, B.A., Che, S., Hower, D.R., et al: ‘Quickrelease: a throughput-oriented approach to release consistency on GPUs’. Int. Symp. High Performance Computer Architecture (HPCA), Orlando, FL, USA, 2014, pp. 189200.
    76. 76)
      • 76. Singh, I., Shriraman, A., Fung, W.W., et al: ‘Cache coherence for GPU architectures’. Int. Symp. High Performance Computer Architecture (HPCA), Shenzhen, People's Republic of China, 2013, pp. 578590.
    77. 77)
      • 77. Koukos, K., Ros, A., Hagersten, E., et al: ‘Building heterogeneous unified virtual memories (UVMs) without the overhead’, Trans. Archit. Code Optim., 2016, 13, (1), pp. 1.11.21.
    78. 78)
      • 78. Zheng, T., Nellans, D., Zulfiqar, A., et al: ‘Towards high performance paged memory for GPUs’. Int. Symp. High Performance Computer Architecture (HPCA), Barcelona, Spain, 2016, pp. 345357.
    79. 79)
      • 79. NVIDIA pascal architecture’. Available at: https://www.nvidia.com/en-us/datacenter/pascal-gpu-architecture/, accessed September 2019.
    80. 80)
      • 80. Ganguly, D., Zhang, Z., Yang, J., et al: ‘Interplay between hardware prefetcher and page eviction policy in CPU-GPU unified virtual memory’. Int. Symp. Computer Architecture (ISCA), Phoenix, AZ, USA, 2019, pp. 112.
    81. 81)
      • 81. Rao, J.N., Sundaresan, M.: ‘Memory sharing via a unified memory architecture’. US Patent Application 16 /219,093, 2016.
    82. 82)
      • 82. Jones, S., Vivek, K., Jaroszynski, P., et al: ‘Unified memory systems and methods’, US Patent Application 16/237,010, 2018.
    83. 83)
      • 83. Kanev, S., Darago, J.P., Hazelwood, K., et al: ‘Profiling a warehouse-scale computer’. Int. Symp. Computer Architecture (ISCA), Portland, OR, USA, 2015, pp. 158169.
    84. 84)
      • 84. Dean, J., Ghemawat, S.: ‘Mapreduce: simplified data processing on large clusters’, Commun. ACM, 2008, 51, (1), pp. 107113.
    85. 85)
      • 85. Zhang, C., Li, P., Sun, G., et al: ‘Optimizing fpga-based accelerator design for deep convolutional neural networks’. Proc. Int. Symp. Field-Programmable Gate Arrays, Monterey, CA, USA, 2015, pp. 161170.
    86. 86)
      • 86. Williams, S., Waterman, A., Patterson, D.: ‘Roofline: an insightful visual performance model for multicore architectures’, Commun. ACM, 2009, 52, (4), pp. 6576.
    87. 87)
      • 87. Hao, Y., Fang, Z., Reinman, G., et al: ‘Supporting address translation for accelerator-centric architectures’. Int. Symp. High Performance Computer Architecture (HPCA), Austin, Texas, USA, 2017, pp. 3748.
    88. 88)
      • 88. Wei, X., Liang, Y., Cong, J.: ‘Overcoming data transfer bottlenecks in FPGA-based DNN accelerators via layer conscious memory management’. Proc. Design Automation Conf (DAC), Las Vegas, NV, USA, 2019, pp. 16.
    89. 89)
      • 89. Sabour, S., Frosst, N., Hinton, G.E.: ‘Dynamic routing between capsules’. Advances in Neural Information Processing Systems, 2017, pp. 38563866.
    90. 90)
      • 90. Marchisio, A., Shafique, M.: ‘Capstore: energy-efficient design and management of the on-chip memory for capsulenet inference accelerators’, ArXiv preprint ArXiv:190201151, 2019.
    91. 91)
      • 91. OpenCAPI consortium sc17 forum’. Available at: https://opencapi.org/wpcontent/uploads/2016/09/OpenCAPI-Overview.10.14.16.pdf, accessed September 2019.
    92. 92)
      • 92. Stuecheli, J., Blaner, B., Johns, C., et al: ‘CAPI: a coherent accelerator processor interface’, IBM J. Res. Dev., 2015, 59, (1), pp. 7.17.7.
    93. 93)
      • 93. An introduction to the intel quickpath interconnect’. Available at: https://www.intel.in/content/dam/doc/white-paper/quick-path-interconnectintroduction-paper.pdf, accessed September 2019.
    94. 94)
      • 94. Abdelrahman, T.S.: ‘Accelerating K-means clustering on a tightly-coupled processor FPGA heterogeneous system’. Int. Conf. Application-specific Systems, Architectures and Processors (ASAP), London, England, 2016, pp. 176181.
    95. 95)
      • 95. Venkata, S.K., Ahn, I., Jeon, D., et al: ‘SD-VBS: the San Diego vision benchmark suite’. Int. Symp. Workload Characterization (IISWC), Austin, TX, USA, 2009, pp. 5564.
    96. 96)
      • 96. Stratton, J.A., Rodrigues, C., Sung, I.J., et al: ‘Parboil: a revised benchmark suite for scientific and commercial throughput computing’, Center Reliable High-Performance Comput., 2012, 17, (2), pp. 112.
    97. 97)
      • 97. Xu, S., Wang, Y., Han, Y., et al: ‘PIMCH: cooperative memory prefetching in processing-in-memory architecture’. Proc. Asia and South Pacific Design Automation Conf., Jeju Island, Republic of Korea, 2018, pp. 209214.
    98. 98)
      • 98. Jeong, T., Choi, D., Han, S., et al: ‘A study of data layout in multi-channel processing-in-memory architecture’. Proc. Int. Conf. Software and Computer Applications, Kuatan, Malaysia, 2018, pp. 134138.
    99. 99)
      • 99. Dai, G., Huang, T., Chi, Y., et al: ‘Graph: a processing-in-memory architecture for large-scale graph processing’, Trans. Comput.-Aided Des. Integr. Circuits Syst., 2019, 38, (4), pp. 640653.
    100. 100)
      • 100. Mutlu, O., Ghose, S., Ausavarungnirun, R.: ‘Recent advances in DRAM and flash memory architectures’, ArXiv preprint ArXiv:180509127, 2018.
    101. 101)
      • 101. Yu, S., Chen, P.Y.: ‘Emerging memory technologies: recent trends and prospects’, Solid-State Circuit Mag., 2016, 8, (2), pp. 4356.
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-cdt.2019.0092
Loading

Related content

content/journals/10.1049/iet-cdt.2019.0092
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
This is a required field
Please enter a valid email address