What the job involves

• We are seeking a motivated and innovative Machine Learning Intern to join our team
• The intern will work on developing a dynamic Key-Value (KV) cache solution for Large Language Model (LLM) inference, aimed at enhancing memory utilization and execution efficiency on D-Matrix hardware
• This project will involve modeling at the PyTorch graph level to enable efficient, torch-native support for KV-Cache, addressing limitations in current solutions
• Research and analyze existing KV-Cache implementations used in LLM inference, particularly those utilizing lists of past-key-values PyTorch tensors
• Investigate “Paged Attention” mechanisms that leverage dedicated CUDA data structures to optimize memory for variable sequence lengths
• Design and implement a torch-native dynamic KV-Cache model that can be integrated seamlessly within PyTorch
• Model KV-Cache behavior within the PyTorch compute graph to improve compatibility with torch.compile and facilitate the export of the compute graph
• Conduct experiments to evaluate memory utilization and inference efficiency on D-Matrix hardware
• Develop an efficient support system for KV-Cache on D-Matrix hardware
• Create a torch-level modeling framework for dynamic KV-Cache
• Ensure compatibility of the KV-Cache model with torch.compile and other PyTorch features for optimized graph export

Benefits

• Equity
• Health care
• Flexible time-off
• Paid paternity leave
• 401k retirement plan
• Remote & hybrid working model
• Work with world-leading engineers and researchers
• Peer coaching & development
• Team-building activities
• Happy hours
• Free food and snacks
• On-site fitness center